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CLAIMS 



[Claim(s)] 

[Claim 1] A pixel detection means to be the Image processing system which extracts 
the background image which is an image of the background from a series of images, 
and to detect the pixel which is in the same location spatially where alignment of the 
background of each of that screen is performed from said a series of images, The 
number means of call meters which carries out counting of the frequency of the pixel 
value of said pixel which is in the same location spatially, A background-image storage 
means by which the rate of said frequency to the number of said pixel which is in the 
same location spatially memorizes the pixel value beyond a predetermined value as a 
pixel value of the background pixel which constitutes said background image, The 
image processing system characterized by including a background escape means to 
extend said background image memorized by said background-image storage means, 
based on said background pixel the pixel value was remembered to be. 
[Claim 2] Said background-image storage means is an image processing system 
according to claim 1 with which said rate of the highest frequency of a pixel value is 
characterized by memorizing the pixel value beyond said predetermined value as a 
pixel value of the background pixel which constitutes said background image. 
[Claim 3] Said background escape means is an image processing system according to 
claim 1 characterized by calculating the pixel value of said background pixel the pixel 
value is not remembered to be based on said background pixel the pixel value close to 
it is remembered to be, and writing in said background-image storage means. 
[Claim 4] Said background escape means is an image processing system according to 
claim 1 characterized by calculating the pixel value of the background [ of not 
registering ] pixel which is said background pixel the pixel value is not remembered to 
be based on a continuity with the background [ of having registered ] pixel which is 
said background pixel the pixel value close to it is remembered to be, and writing in 
said background-image storage means. 

[Claim 5] The pixel value which has the rate of the frequency of the pixel value of said 
background [ of not registering ] pixel, and the pixel of said a series of images which a 
space target has in the same location in predetermined ranking said background 
escape means When it has the pixel value and continuity of said background [ of 
having registered ] pixel which adjoin said background [ of not registering ] pixel. The 
Image processing system according to claim 4 characterized by for the rate of said 



JP11258473 



frequency of having the pixel value and continuity of said background [ of having 
registered ] pixel calculating the pixel value of said background [ of not registering ] 
pixel, and writing in said background-image storage means based on the pixel value 
which is within predetermined order. 

[Claim 6] The pixel detection step which is the image-processing approach of 
extracting the background image which is an image of the background from a series of 
images, and detects the pixel which is in the same location spatially where alignment 
of the background of each of that screen is performed from said a series of images. 
The number step of call meters which carries out counting of the frequency of the 
pixel value of said pixel which is in the same location spatially. The rate of said 
frequency to the number of said pixel which is in the same location spatially the pixel 
value beyond a predetermined value as a pixel value of the background pixel which 
constitutes said background image The image-processing approach characterized by 
including the background-image storage step which a background-image storage 
means to memorize said background image is made to memorize, and the background 
escape step which extends said background image memorized by said 
background-image storage means based on said background pixel the pixel value was 
remembered to be. 

[Claim 7] The program for performing the image processing which extracts the 
background image which is an image of the background from a series of images The 
pixel detection step which is the medium which a computer is made to perform and 
detects the pixel which is in the same location spatially from said a series of images 
where alignment of the background of each of that screen is performed. The number 
step of call meters which carries out counting of the frequency of the pixel value of 
said pixel which is in the same location spatially. The rate of said frequency to the 
number of said pixel which is in the same location spatially the pixel value beyond a 
predetermined value as a pixel value of the background pixel which constitutes said 
background image The background-image storage step which a background-image 
storage means to memorize said background image is made to memorize. The medium 
which makes said computer execute the program characterized by including the 
background escape step which extends said background image memorized by said 
background-image storage means based on said background pixel the pixel value was 
remembered to be. 
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[Detailed Description of the Invention] 
[0001] 

[Field of the Invention] This invention relates to a medium at the image processing 
system and the image-processing approach of enabUng it to extract the background 
especially displayed on the image processing system and the image-processing approach, 
and the list by a series of images about the mediiim with a sufEicient precision, and a Ust. 
[0002] 

[Description of the Prior Art] Conventionally, it considers as the approach of carrying 
out compression coding of the image, for example, there is object coding. In object coding, 
the foreground and background are extracted from a series of images (one scene or 
image of 1 cut) (for example, frame from a certain scene change to the next scene change 
etc.). That is, when a series of images consist of N frames, the background (in a video 
camera, if this background is panning or the scene photoed by carrying out chill TINGU, 
it will serve as an oblong or longwise image) of the whole 1 scene displayed with that N 
frame, and the foreground which exists in common in the N frame are extracted. And 
decode of the data obtained by object coding is performed by arranging a foreground for 
a background. 
[0003] 

[Problem(s) to be Solved by the Invention] By the way, in coding which extracts a 
foreground and a background, the precision which extracts a background influences the 
image quality of the decode image greatly like above-mentioned object coding, for 
example. 

[0004] This invention is made in view of such a situation, and enables it to extract a 

background with a sufficient precision from a series of images. 

[0005] 

[Means for Solving the Problem] A pixel detection means by which the image processing 
system of this invention detects the pixel which is in the same location spatially from a 
series of images where alignment of the background of each of that screen is performed. 
The number means of call meters which carries out counting of the frequency of the 
pixel value of the pixel which is in the same location spatially, A background-image 
storage means by which the rate of frequency to the number of the pixel which is in the 
same location spatially memorizes the pixel value beyond a predetermined value as a 
pixel value of the background pixel which constitutes a background image, It is 
characterized by including a background escape means to extend the background image 
memorized by the background-image storage means, based on the background pixel the 
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pixel value was remembered to be. 

[0006] The rate of the highest frequency of a pixel value can make a background-image 
storage means memorize the pixel value beyond a predetermined value as a pixel value 
of the background pixel which constitutes a background image. 

[0007] A background escape means can be made to be asked for the pixel value of the 
background pixel the pixel value is not remembered to be based on the background pixel 
close to it the pixel value is remembered to be, and it can be made to write in a 
background-image storage means. Moreover, a background escape means can be made 
to be asked for the pixel value of the background [ of not registering ] pixel which is a 
background pixel the pixel value is not remembered to be based on a continuity with the 
background [ of having registered ] pixel close to it which is a background pixel the pixel 
value is remembered to be, and it can be made to write in a background-image storage 
means. For a background escape means, furthermore, the pixel value which has the rate 
of the frequency of the pixel value of a background [ of not registering ] pixel, and the 
pixel of a series of images which a space target has in the same location in 
predetermined ranking When it has the pixel value and continuity of a background [ of 
having registered ] pixel which adjoin a background [ of not registering ] pixel, the pixel 
value of a background [ of not registering ] pixel can be made to be calculated, and it can 
be made to write in a background-image storage means based on the pixel value which 
has the pixel value and continuity of a background [ of having registered ] pixel and 
which has the rate of frequency within predetermined order. 

[0008] The pixel detection step to which the image -processing approach of this invention 
detects the pixel which is in the same location spatially from a series of images where 
alignment of the background of each of that screen is performed. The number step of 
call meters which carries out counting of the frequency of the pixel value of the pixel 
which is in the same location spatially. The rate of frequency to the number of the pixel 
which is in the same location spatially the pixel value beyond a predetermined value as 
a pixel value of the background pixel which constitutes a background image It is 
characterized by including the background-image storage step which a 
background-image storage means to memorize a background image is made to 
memorize, and the background escape step which extends the background image 
memorized by the background-image storage means based on the background pixel the 
pixel value was remembered to be. 

[0009] The program which the medium of this invention makes a computer execute The 
pixel detection step which detects the pixel which is in the same location spatially from 
a series of images where alignment of the background of each of that screen is 
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performed, The number step of call meters which carries out counting of the frequency 
of the pixel value of the pixel which is in the same location spatially, The rate of 
frequency to the number of the pixel which is in the same location spatially the pixel 
value beyond a predetermined value as a pixel value of the background pixel which 
constitutes a background image It is characterized by including the background-image 
storage step which a background-image storage means to memorize a background 
image is made to memorize, and the background escape step which extends the 
background image memorized by the background-image storage means based on the 
background pixel the pixel value was remembered to be. 

[OOlO] The pixel which is in the same location spatially where alignment of the 
background of each of that screen is performed in the image processing system of this 
invention and the image-processing approach, and a list from a series of images in a 
medium is detected, and counting of the frequency of the pixel value of the pixel which 
is in the same location spatially is carried out. And based on the background pixel the 
rate of frequency to the number of the pixel which is in the same location spatially was 
remembered to be by background-image storage means to memorize a background 
image, as a pixel value of the background pixel from which the pixel value beyond a 
predetermined value constitutes a background image, and the pixel value was 
remembered to be, the background image memorized by the background-image storage 
means is extended. 
[0011] 

[Embodiment of the Invention] Drawing 1 shows the example of a configuration of the 
gestalt of 1 operation of the image transmission equipment which appUed this invention. 
[0012] The digital image data as an image (here, it considers as a dynamic image) set as 
the object of coding photoed with the video camera etc. are suppUed, the image is 
encoded by the encoder 1 and the coded data obtained as a result is outputted to it there. 
This coded data is transmitted through the transmission media 3, such as a satellite 
circuit, and a ground wave, a CATV (Cable Television) network, the Internet, ISDN 
(Integrated Service Digital Network), or is recorded on the record media 4, such as an 
optical disk, and a magneto-optic disk, a magnetic disk, a magnetic tape, a phase 
change disk. 

[0013] The coded data transmitted through a transmission medium 3 or the coded data 
reproduced from a record medium 4 is supplied to a decoder 2, in a decoder 2, the coded 
data supplied there is decoded, and the decode image obtained as a result is supplied 
and displayed on the monitor which is not illustrated, for example. 
[0014] The above image transmission equipment is applicable to the equipment which 
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transmits and receives an image in the distant location, for example, the eqmpment 
which performs record playback of an image. 

[0015] Next, drawing 2 shows the example of a configuration of the encoder 1 of drawing 
1 . 

[0016] The are recording section 11 consists of semiconductor memory, a magnetic disk, 
etc., and stores temporarily the image data supphed to an encoder 1. Here, image data 
is inputted and memorized by the are recording section 11, for example per frame of a 
series of images, such as from a certain scene change until [ of a degree ] a scene change. 
In addition, as shown in drawing 3 , the image data of the N frame which becomes by 
the 1st frame thru/or the Nth frame shall be memorized by the are recording section 11 
as a series of images here, for example. 

[0017] If the image of the N frame as a series of images is memorized by the are 
recording section 11, the camera motion detecting element 12 will read the image of the 
N frame from the are recording section 11, and will detect the camera motion vector 
showing the direction and magnitude of a camera motion in each frame. 
[0018] That is, the camera motion detecting element 12 sets the camera motion vector 
vl of the 1st frame as 0 (= (O O)) in the system of coordinates (suitably henceforth 
standard coordinates) which make the right or down [ a top to ] a x axis or the yaxis 
from the left, respectively while making a zero the point at the upper left of the 1st 
frame, as shown in drawing 4 (A). And about the 2nd frame thru/or the Nth frame, the 
camera motion detecting element 12 searches for the coordinate (x y) of the criteria seat 
table system in which the point at the upper left of the n-th frame is located as the 
camera motion vector vn of the n-th firame, when alignment of the background of each 
frame is performed. 

[0019] After the camera motion detecting element 12 sets the camera motion vector vl 
of the 1st frame as 0, as shown in drawing 4 (B), in standard coordinates, it asks for the 
location of the 1st frame and the 2nd frame whose backgrounds correspond, and, 
specifically, searches for the coordinate of the point at the upper left of [ in the location ] 
the 2nd frame as the camera motion vector v2. Furthermore, as shown in drawing 4 (C), 
in standard coordinates, the camera motion detecting element 12 asks for the location of 
the 1st frame which performed alignment of a background and the 2nd frame, and the 
3rd frame whose backgrounds correspond, and searches for the coordinate of the point 
at the upper left of [ in the location ] the 3rd frame as the camera motion vector v3. 
[0020] Hereafter, the camera motion detecting element 12 calculates the camera motion 
vector v4 of the 4th frame thru/or the Nth frame thru/or vN similarly. 
[002 1] In addition, in order to simplify explanation, as a camera motion, only a motion 
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of a perpendicular direction is considered and let revolutions be level and the thing 
which is not considered here. However, this invention can be applied even when a 
camera motion has a revolution. 

[0022] The camera motion vector vl of the 1st frame thru/or the Nth frame as a series of 
images detected by the camera motion detecting element 12 as mentioned above thrWor 
vN are supplied to the camera motion vector storage section 13, and is memorized. 
[0023] When the camera motion vector vl thru/or vN are memorized, in the camera 
motion vector storage section 13 the background extract section 14 While reading the 
camera motion vector vl thru/or vN from the camera motion vector storage section 13 
Read the image data of the 1st frame thru/or the Nth frame from the are recording 
section 11, and by performing alignment of the background of the 1st frame thru/or the 
Nth frame based on the camera motion vector vl thru/or vN the 1st frame thru/or that 
the Nth frame background of the whole (this background - for example, a series of 
images ■- a video camera ■■ panning -■ or if chill TINGU is carried out and a photograph 
is taken, it will become an oblong or longwise image) (suitably henceforth a whole 
background) are extracted. The whole background extracted in the background extract 
section 14 is supplied to the background memory 15, and is memorized. 
[0024] In the background memory 15, if a whole background is memorized, the 
foreground coding section 16 detects the background of each frame memorized by the 
are recording section 1 1 among the whole background based on the camera motion 
vector of each frame memorized by the camera motion vector storage section 13, will be 
subtracting the background of each of that detected frame from the image of each frame, 
and will extract the foreground of each frame. Furthermore, the foreground coding 
section 16 encodes the foreground of each frame, and outputs the coding result to 
MUX(multiplexer) 17. 

[0025] From the foreground coding section 16, if the coding result of a foreground is 
received, MUX17 will multiplex the camera motion vector memorized by the camera 
motion vector storage section 13 and the whole background memorized by the 
background memory 15 to the coding result of the foreground, and wiU output the 
multiplexing result to it as coded data. 

[0026] In an encoder 1, image data is encoded per a series of images as mentioned above. 
[0027] Next, drawing 5 shows the example of a configuration of the camera motion 
detecting element 12 of drawing 2 . 

[0028] A series of images memorized by the are recording section 11 ( drawing 2 ) are 
supplied to the center-of-gravity calculation section 21 per frame, and the 
center-of-gravity calculation section 21 asks it for a center of gravity which is later 
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mentioned about each frame. Furthermore, the center -of- gravity calculation section 21 
sets up the range (suitably henceforth the motion detection range) used for detecting 
the camera motion vector of the attention frame currently observed to the are recording 
image mentioned later memorized in the are recording image memory 24, and also 
searches for the center of gravity of the motion detection range. The center of gravity of 
the attention frame called for in the center-of-gravity calculation section 21 and the 
motion detection range is supplied to the vector detecting element 22. 
[0029] The vector detecting element 22 is supplied to the write-in control section 23 
while it detects the camera motion vector of an attention frame and suppUes it to the 
camera motion vector storage section 13 ( drawing 2 ) based on the center of gravity of 
the attention frame supplied from the center-of-gravity calciilation section 21, and the 
motion detection range. 

[OOSO] The write-in control section 23 controls the address with which the are recording 
image memory 24 memorizes the image data of an attention frame based on the camera 
motion vector from the vector detecting element 22. From the are recording section 11 
( drawing 2 ), the are recording image memory 24 reads the image data of an attention 
frame, and memorizes it to the address specified by the write-in control section 23. 
[0031] Next, with reference to drawing 6 , the camera motion detection processing 
performed in the camera motion detecting element 12 of drawing 5 in which a camera 
motion vector is detected is explained. 

[0032] The point at the upper left of [ in the condition of having performed alignment of 
each frame and having performed the alignment ] each frame is detected as a camera 
motion vector of each frame so that the center of gravity of each frame may be in 
agreement fundamentally paying attention to the center of gravity of an image moving 
the camera motion detection processing which the camera motion detecting element 12 
performs by camera motion. 

[0033] That is, if the n-th frame is now used as an attention frame, the image in the 
condition of having performed alignment of the background of each frame in the are 
recording image memory 24, and having laid the 1st thru/or the image data of the n-lst 
frame which is a frame to the frame in front of an attention frame on top of it in the 
sequence (are recording image) is memorized. 

[0034] In this case, the center-of-gravity calculation section 21 searches for the center of 
gravity cn of the n-th frame which is an attention frame, as shown in drawing 6 (A). 
Furthermore, as shown in drawing 6 (B), the center-of-gravity calculation section 21 
makes the range which includes the n-lst frame in front of one of the attention frames 
in the are recording image memorized in the are recording image memory 24 the motion 
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detection range, and searches for the center of gravity c of the motion detection range. 
Here, as motion detection range, the range where only the predetermined number of 
pixels is large is set up in each direction of four directions of the n-lst frame, for 
example. 

[0035] By the vector detecting element 22, if the center of gravity cn of an attention 
frame and the center of gravity c of the motion detection range are searched for, as 
shown in drawing 6 (C), the location of the point at the upper left of [ in the condition 
that the center of gravity cn of an attention frame was in agreement with the center of 
gravity c of the motion detection range ] an attention frame will be called for, and the 
coordinate of the location will be outputted as a camera motion vector vn of the n-th 
frame which is an attention frame. 

[0036] That is, considering the n-th frame as an attention frame, in asking for the 
camera motion vector vn, it has already found the camera motion vector of a before [ one 
of them ]. Then, as shown in drawing 6 (C), while expressing the location on the basis of 
the point at the upper left of [ of the center of gravity c of the motion detection range ] 
the n-lst frame with vector v'c Supposing it expresses with vector v'cn the location on 
the basis of the point at the upper left of [ of the center of gravity cn of the n-th frame 
which is an attention frame / the ] the n-th frame The coordinate in the standard 
coordinates of the location of the point at the upper left of [ in the condition that the 
center of gravity c of the motion detection range and the center of gravity cn of an 
attention frame were in agreement ] an attention frame serves as the motion vector vn 
of an attention frame. And it can ask for this camera motion vector vn by subtracting 
vector v'cn which adds vector v'c showing the location of the center of gravity c of the 
motion detection range to motion vector vn-1 in front of one attention frame of the n-lst 
frame, and expresses the location of the center of gravity cn of an attention frame to it 
further. That is, it can ask for the camera motion vector vn of an attention frame by 
calculating formula vn=vn-l-^v'c-v'cn. 

[0037] After the camera motion vector vn of an attention frame is called for as 
mentioned above, in the write-in control section 23, the write-in address for writing in 
the image data of the attention frame in the are recording image memory 24 is 
controlled based on the camera motion vector vn. Namely, thereby, in standard 
coordinates, the image data of an attention frame is written in the point shown by the 
camera motion vector vn in the form to overwrite, and the image obtained as a result of 
the writing is used for it by using the following frame [ n-i-lst ] as an attention frame in 
the are recording image memory 24 as an are recording image at the time of detecting 
the camera motion vector vn-H so that the point of the upper left may be located. 
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[0038] Next, with reference to the flow chart of drawing 7 , the camera motion detection 
processing in the camera motion detecting element 12 of drawing 5 is explained further. 
[0039] While reading appearance of the 1st frame of a series of images memorized by 
the are recording section 1 1 is carried out as an attention frame and it is first supphed 
to the center-of-gravity calculation section 21, the storage value of the are recording 
image memory 24 is cleared. 

[0040] And in the center-of-gravity calculation section 21, it is judged in step Si 
whether there is any 1st attention frame. In step SI, when judged with there being the 
1st attention frame, it progresses to step S2, and the vector detecting element 22 sets up 
0 as the camera motion vector vl, outputs it to the camera motion vector storage section 
13 and the write-in control section 23, and progresses to step S6. 
[0041] Based on the camera motion vector from the vector detecting element 22, the 
write-in control section 23 controls the write-in address in the are recording image 
memory 24 by step S6, and, thereby, writes an attention frame in the are recording 
image memory 24 at it. That is, in now, there is the 1st attention frame, and since the 
camera motion vector vl is 0, the image data of the 1st frame is written in in the are 
recording image memory 24 so that the point of the upper left may be located in the zero 
in standard coordinates. 

[0042] Then, it progresses to step S7, and when judged with it being judged whether 
there is any following frame which constitutes a series of images, and occurring by the 
are recording section 11, reading appearance of the following frame is newly carried out 
to it as an attention frame, and it is supplied to it at the center-of-gravity calculation 
section 21. And the same processing is repeated by step SI return and the following. 
[0043] When it is judged with there not being the 1st attention frame in step SI on the 
other hand (i.e., when it is either the 2nd frame or thru/or the Nth frame), it progresses 
to step S3, and in the center-of-gravity calculation section 21, center-of-gravity 
calculation processing in which the center of gravity of an attention frame is searched 
for is performed, and it progresses to step S4. In step S4, in the center-of-gravity 
calculation section 21, the motion detection range to an attention frame is set up into 
the are recording image memorized in the are recording image memory 24, 
center-of-gravity calculation processing in which the center of gravity of the motion 
detection range is searched for is performed, and it progresses to step S5. 
[0044] At step S5, in the vector detecting element 22, as drawing 6 explained, the 
camera motion vector of an attention frame is called for, and it is outputted to the 
camera motion vector storage section 13 and the write-in control section 23 from the 
center of gravity of the attention frame called for in the center-of-gravity calculation 
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section 21, and the center of gravity of the motion detection range. 

[0045] And as progressed and mentioned above to step S6, in the write-in control section 
23, based on the camera motion vector from the vector detecting element 22, the 
write-in address in the are recording image memory 24 is controlled, and, thereby, an 
attention frame is written in the are recording image memory 24. That is, in the are 
recording image memory 24, in standard coordinates, the image data of an attention 
frame is written in the point shown by the camera motion vector of an attention frame 
so that the point of the upper left may be located (overwritten). 

[0046] Then, as progressed and mentioned above to step S7, when it is judged whether 
the following frame which constitutes a series of images is in the are recording section 
11 and it is judged with there being nothing (i.e., when the 1st which constitutes a 
series of images thru/or the camera motion vector of each Nth frame are called for), 
camera motion detection processing is ended. 

[0047] Next, with reference to the flow chart of drawing 8 , the processing 
(center-of-gravity calculation processing of an attention frame) which the 
center-of- gravity calculation section 21 performs is explained in full detail in step S3 of 
drawing 7 . 

[0048] First, in step Sll, to Variables X or Y, the number of pixels beside an attention 
frame (horizontal direction) or the vertical (perpendicular direction) number of pixels is 
set, respectively, and progresses to step S12. 

[0049] -1 as initial value is set to the variable y which expresses the y-coordinate of each 
pixel of an attention frame with step S12, it progresses to step S22, and the increment of 
the variable y is carried out only for 1 . 

[0050] Here, the coordinate of each pixel which constitutes an attention frame makes 
the pixel on the leftmost a zero, shall take a x axis or the y-axis from the left from a top 
to the right or down, respectively, and shall express it. 

[0051] Then, it progresses to step S14 and it is judged whether Variable y is under the 
number Y of pixels of the length of an attention frame. In step S14, when judged with 
Variable y being under Y, it progresses to step S15, and -1 as initial value is set to the 
variable x showing the x-coordinate of each pixel of an attention frame, and it 
progresses to step S16. 

[0052] At step S16, the increment only of 1 is carried out. Variable x progresses to step 
S17, and it is judged whether Variable x is under the number X of pixels beside an 
attention frame. In step S17, when judged with Variable x not being under X, the same 
processing is repeated by step S13 return and the following. 

[0053] Moreover, in step S17, when judged with Variable x being under X, it progresses 
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to step S18, the pixel p (x y) in a coordinate (x y) is made into an attention pixel, and the 
attention pixel is classified into either of the level set up beforehand based on the pixel 
value. 

[0054] That is, with the gestalt of this operation, the range of the value which can be 
taken as a pixel value is beforehand divided into some range. And if the range of the 
value which can be taken as a now, for example, pixel, value shall be divided into K 
range and this K range shall be called level 1, 2, K in an order from the range where 
a pixel value is small, at step S18, an attention pixel will be classified according to 
whether a pixel value belongs to level 1 thru/or which range of the K. 
[0055] Furthermore, the level classification result of an attention pixel is registered into 
a level table at step SIS. 

[0056] The center-of-gravity calculation section 21 Namely, among those, the frequency 
fk of the pixel which belongs to the memory (not shown) to harbor at the level k about 
each level k (k= 1, 2, K) as shown in drawing 9 , When the level table which matched 
integrated value sigmaxk of the x-coordinate of the pixel belonging to level k and 
integrated value sigmayk of a ycoordinate is memorized, for example, the pixel value of 
an attention pixel belongs to level k While only 1 increments the frequency fk about the 
level k in a level table, the x-coordinate or ycoordinate of an attention pixel is added to 
integrated value sigmaxk of an x-coordinate, or integrated value sigmayk of a 
ycoordinate, respectively. 

[0057] In addition, a level table is cleared by 0 whenever processing according to the 
flow chart of drawing 8 is started. 

[0058] And the same processing is repeated by step S16 return and the following. 
[0059] It processes by making into an attention pixel each pixel which constitutes an 
attention frame on the other hand when judged with Variable y not being under Y in 
step S14, and when all the pixels that constitute an attention frame are registered into 
a level table, it progresses to step S19 and the center of gravity of the pixel belonging to 
each level of a level table is searched for. That is, the coordinate (sigmaxk/fk, 
sigmayk/fk) as which the division of integrated value sigmaxk of the x-coordinate in 
each level k of a level table or each integrated value sigmayk of a y-coordinate is done in 
the frequency fk, and it is expressed in step S19 with the division value is searched for 
as a center of gravity of the pixel belonging to each level k. 

[0060] and the step S20 -- progressing - level 1 thrvi/or K - it is alike, respectively, and 
the center of gravity of the center of gravity of the pixel which belongs is searched for as 
a center of gravity of the whole pixel which constitutes an attention frame, and carries 
out a return. 
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[0061] namely, - step S20 - level 1 thru/or K ■- it is alike, respectively, and the 
weighting average which makes weight the frequency fl thru/or fK is calculated about 
the center of gravity of the pixel which belongs, and it is outputted as a center of gravity 
of the whole pixel where the weighting average constitutes an attention frame. 
[0062] Next, with reference to the flow chart of drawing 10 , the processing 
(center-of-gravity calculation processing of the motion detection range) which the 
center-of-gravity calculation section 21 performs is explained in full detail in step S4 of 
drawing 7 . 

[0063] First, the motion detection range is set up in step S31. That is, at step S31, the 
range where the frame in front of one attention frame was written in is detected from 
the are recording image memorized in the are recording image memory 24. 
Furthermore, at step S31, the detected range can extend only the predetermined 
number of pixels for example, in the direction of four directions, respectively, and the 
range which was able to be extended is set up as motion detection range. 
[0064] And it progresses to step S32, and to Variables X or Y, the number of pixels 
beside the motion detection range or the vertical number of pixels is set, respectively, 
and progresses to step S33. 

[0065] Then, in step S33 thru/or S41, step S12 of drawing 8 thru/or the respectively 
same processing as the case in S20 are performed, and thereby, the center of gravity of 
the whole pixel which constitutes the motion detection range is searched for, and carries 
out a return. 

[0066] In the camera motion detecting element 12, the predetermined range containing 
the frame in front of one of the attention frames in an are recording image is set up as 
motion detection range. As mentioned above, the center of gravity of the motion 
detection range, Since the center of gravity of an attention frame is computed, and it 
asks for the camera motion vector of an attention frame based on those centers of 
gravity and was made to repeat writing an attention frame in an are recording image 
based on the camera motion vector As compared with the case where the so-called block 
matching is performed, it can ask for a camera motion vector simply. 
[0067] In addition, although the center of gravity of an attention frame is searched for 
and the center of gravity of the motion detection range was searched for in step S4 after 
that in step S3 with the gestalt of operation of drawing 7 , you may ask for whichever 
first and the center of gravity of an attention frame and the center of gravity of the 
motion detection range can also be made to ask simultaneously. 

[0068] Next, although it asked for each pixel which constitutes the attention frame for 
the center of gravity of an attention frame with the gestalt of operation of drawing 8 by 
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classifying into either of some level according to the pixel value (suitably henceforth a 
level classification) In addition to this, the center of gravity of an attention frame for 
example, each pixel which constitutes the attention frame It is also possible to carry out 
based on some pixels (for self to be included and for it not to be necessary to include) 
around it by classifying into either of some classes (suitably henceforth a class 
classification). 

[0069] Here, a class classification is explained briefly. A now, for example, attention, 
pixel shall constitute the tap (suitably henceforth a class tap) used for the class 
classification of an attention pixel from the 4 pixels which adjoins vertically and 
horizontally, respectively, and own a total of 5 pixels of an attention pixel. If a pixel 
value shall be expressed by 1 bit in this case (it becomes the value of either 0 or l), an 
attention pixel can be classified into the pattern of 32 (= (2l) 5) according to the pixel 
value which 5 pixels of the class tap constituted about that attention pixel can take. It is 
such a pattern part injury class classification, and an attention pixel will be classified 
into either of the classes of 32 when it is now. 

[0070] In addition, if the class tap which becomes by 5 pixels as mentioned above is 
constituted and a class classification is generally performed when 8 bits is assigned to 
the pixel although about 8 bits is assigned to a pixel, the number of classes will turn 
into a huge number called (28) 5. 

[0071] Then, to the class tap constituted about an attention pixel, L bit ADRC (Adaptive 
Dynamic Range Coding) processing is carried out, and a class classification can be 
performed based on the class tap after the ADRC processing. 

[0072] Here, in L bit ADRC processing, for example, Maximum MAX and the minimum 
value MIN of a pixel value of a pixel which constitute a class tap are detected, 
DR=MAX-MIN is used as the local dynamic range of a set, and the pixel which 
constitutes a class tap is re-quantized by L bits based on this dynamic range DR. That is, 
out of the pixel value of the pixel which constitutes a class tap, the minimum value MIN 
is subtracted and the division (quantization) of the subtraction value is done by DR/2L. 
Therefore, when L bit ADRC processing of the class tap is carried out, as compared with 
the case where a class classification is performed without carrying out ADRC processing 
of the class tap, the number of classes can be decreased by the pixel value of each pixel 
which constitutes the class tap being made into L bits, and considering as a value 
smaller than the number of bits to which L was assigned by the pixel. 
[0073] In addition, moreover, when the pixel of the upper bed of a frame, a soffit, a left 
end, or a right end turns into an attention pixel, since the pixel which adjoins the left or 
the right the bottom does not exist, it shall constitute a class tap, assuming it to be that 
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to which the same frame turns up and exists in a frame upside, the bottom, left-hand 
side, or right-hand side in this case. 

[0074] Next, by carrying out the class classification of each pixel which constitutes an 
attention frame with reference to the flow chart of drawing 11 based on the class tap 
about the pixel explains the processing (center-of- gravity calculation processing of an 
attention frame) performed in the center-of- gravity calculation section 21 in the case of 
searching for the center of gravity of an attention frame. 

[0075] In this case, in step S51 thru/or S57, step Sll of drawing 8 thru/or the 
respectively same processing as the case in S17 are performed. And the pixel p (x y) in a 
coordinate (x y) is made into an attention pixel, and a class tap is constituted from step 
S58 corresponding to step SIS of drawing 8 by the attention pixel. And based on the 
class tap, the class classification of the attention pixel is carried out at either of the K 
classes, and the class classification result is registered into a class table. 
[0076] In the gestalt of operation of drawing 11 namely, the center-of-gravity calculation 
section 21 In the memory to build in, for example, the frequency fk of the pixel which 
belongs to the class k about each class k (k= 1, 2, K) as shown in drawing 12 , When 
the class table which matched integrated value sigmaxk of the x-coordinate of the pixel 
belonging to Class k and integrated value sigmayk of a y-coordinate is memorized, for 
example, an attention pixel belongs to Class k While only 1 increments the frequency fk 
about the class k in a class table, the x-coordinate or y-coordinate of an attention pixel is 
added to integrated value sigmaxk of an x-coordinate, or integrated value sigmayk of a 
y-coordinate, respectively. 

[0077] In addition, a class table is cleared by 0 whenever processing according to the 
flow chart of drawing 11 is started like a level table. 

[0078] The same processing is repeated by step S56 return and the following after 
processing of step S58. 

[0079] On the other hand, when judged with Variable y not being under Y in step S54, it 
progresses to step S59 and the center of gravity of the pixel belonging to each class of a 
class table is searched for. That is, the coordinate (sigmaxk/fk, sigmayk/fk) as which the 
division of integrated value sigmaxk of the x-coordinate in each class k of a class table or 
each integrated value sigmayk of a y-coordinate is done in the frequency fk, and it is 
expressed in step S59 with the division value is searched for as a center of gravity of the 
pixel belonging to each class k. 

[0080] and the step S60 " progressing -- a class 1 thru/or K it is alike, respectively, 
and the center of gravity of the center of gravity of the pixel which belongs is searched 
for as a center of gravity of the whole pixel which constitutes an attention frame, and 
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carries out a return. 

[0081] namely, -- step S60 -- a class 1 thru/or K — it is alike, respectively, and the 
weighting average which makes weight the frequency fl thru/or fK is calculated about 
the center of gravity of the pixel which belongs, and it is outputted as a center of gravity 
of the whole pixel where the weighting average constitutes an attention frame. 
[0082] When searching for the center of gravity of an attention frame by carrying out 
the class classification of each pixel which constitutes an attention frame here based on 
the class tap about the pixel, it is desirable to make it ask by the center of gravity of the 
motion detection range also constituting a class tap about each pixel which constitutes 
the motion detection range, and performing a class classification based on the class tap. 
[0083] In addition, in searching for the center of gravity of the motion detection range by 
performing a class classification, in the processing which showed the flow chart of 
drawing 11 , it changes to processing of the step S51, and since it is the same as that of 
the case where processed steps S31 and S32 shown in the flow chart of drawing 10 , and 
also it is shown in drawing 11 , it omits the explanation. 

[0084] Next, although the center of gravity of an attention frame was searched for with 
the gestalt of operation of drawing 11 using the center of gravity of the pixel belonging 
to all the classes that carry out the class classification of each pixel which constitutes an 
attention frame based on the class tap about the pixel, and are obtained as a result, the 
center of gravity of an attention frame can be made to ask only using the center of 
gravity of the pixel which belongs to the specific class obtained as a resiilt offer example, 
a class classification in addition to this. 

[0085] That is, the center of gravity of an attention frame can be asked for the class to 
which the pixel (suitably henceforth an edge pixel) of the part which is an edge belongs 
as a specific class only using the center of gravity of the pixel (edge pixel) belonging to 
the specific class (suitably henceforth an edge class). 

[0086] Then, when asking with reference to the flow chart of drawing 13 only using the 
center of gravity of the edge pixel which belongs to an edge class among the class 
classification results of each pixel which constitutes an attention frame, the processing 
(center-of- gravity calculation processing of an attention frame) performed in the 
center-of-gravity calculation section 21 is explained. 

[0087] Also in this case, step Sll of drawing 8 thru/or the respectively same processing 
as the case in S17 are performed in step S71 thrvi/or S77. And in step S77, when judged 
with Variable x being under X, it progresses to step S78, the pixel p (x y) in a coordinate 
(x y) is made into an attention pixel, and the class classification of the attention pixel is 
carried out Uke the case in step S58 of drawing 11 . 
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[0088] Then, it progresses to step S79 and it is judged [ the class obtained as a result of 
the class classification in step S78 ] for it being an edge class, i.e., an attention pixel, 
whether it is an edge pixel. 

[0089] Here, an attention pixel is classified into either of the 1024 (= (22) 5) classes 
when performing a class classification, after carrying out 2-bit ADRC processing of the 
class tap which consists of 5 pixels which was mentioned above, for example. By the way, 
when 2-bit ADRC processing of the class tap is carried out, the pixel value of the pixel 
which constitutes the class tap turns into a value of the OOB, OIB, lOB, or the llB (B 
expresses that the value arranged before that is a binary number). Therefore, as 
mentioned above, when a class tap consists of 4 pixels [ an attention pixel and / which 
adjoins the four directions of the attention pixel, respectively ] a total of 5 pixels, it sets. 
While the time of the pixel value of 4 pixels which adjoins vertically and horizontally, 
respectively being except OOB while the pixel value of an attention pixel is OOB which is 
the minimum value, and the pixel value of an attention pixel are IIB which is 
maximum When the pixel value of 4 pixels which adjoins vertically and horizontally, 
respectively is except IIB, it is thought that the attention pixel is an edge pixel. 
[0090] That is, as shown in drawing 14 (A), when the pixel value of an attention pixel is 
OOB and the pixel value of 4 pixels which adjoins vertically and horizontally, 
respectively is OIB, lOB, or the IIB, so to speak, the pixel value is a trough (concave) in 
the attention pixel. Moreover, as shown in drawing 14 (B), when the pixel value of an 
attention pixel is IIB and the pixel value of 4 pixels which adjoins vertically and 
horizontally, respectively is OOB, OIB, or the lOB, so to speak, the pixel value is a crest 
(convex) in the attention pixel. Therefore, the attention pixel is an edge pixel when 
shown in drawing 14 . 

[0091] In addition, when the pixel value of an attention pixel is OOB and the pixel value 
of 4 pixels which adjoins vertically and horizontally, respectively is OIB, lOB, or the IIB 
When the pixel value of a **** and an attention pixel is IIB 81 (= 3x3x3x3) passages 
and the pixel value of 4 pixels which adjoins vertically and horizontally, respectively is 
OOB, OIB, or the lOB, there are 81 kinds similarly. Therefore, there is a class (edge 
class) to which an edge pixel belongs 162 (= 81-t-8l) passages. 

[0092] When it is judged with an attention pixel not being an edge pixel by drawing 13 
in return and step S79 (i.e., when the classes of an attention pixel are not any of the 
above-mentioned edge classes which have 162 kinds, either), it returns to step S76. 
[0093] Moreover, in step S79, when judged with an attention pixel being an edge pixel 
(i.e., when the class of an attention pixel is either of the above-mentioned edge classes 
which have 162 kinds), it progresses to step S80 and the class classification result of an 
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attention pixel is registered into a class table. That is, at step S80, while the increment 
of the frequency fk about the class k of an attention pixel of a class table as shown in 
drawing 12 is carried out only for 1, the x-coordinate or y-coordinate of an attention 
pixel is added to integrated value sigmaxk of an x-coordinate, or integrated value 
sigmayk of a y-coordinate, respectively. 

[0094] In addition, in performing a class classification and registering only an edge class 
into a class table only about the attention pixel belonging to an edge class with the 
gestalt of operation of drawing 13 , for example after carrying out 2-bit ADRC 
processing of the class tap which consists of 5 pixels as mentioned above since 
registration to a class table is performed, the number K of classes of a class table is set 
to 162 which is the number of edge classes mentioned above. 

[0095] Since it is classified into either of the 1024 classes as the attention pixel was 
mentioned above when performing a class classification and registering all classes into 
a class table on the other hand after carrying out 2-bit ADRC processing of the class tap 
which consists of 5 pixels, the number K of classes of a class table is set to 1024. 
[0096] Therefore, in the gestalt of operation of drawing 13 , magnitude (capacity) of a 
class table can be made small as compared with the case in the gestalt of operation of 
drawing 11 . 

[0097] The same processing is repeated by step S76 return and the following after 
processing of step S80. 

[0098] On the other hand, when judged with Variable y not being under Y in step S74, it 
progresses to step S81 and the center of gravity of the pixel belonging to each edge class 
of a class table is searched for. That is, the coordinate (sigmaxk/fk, sigmayk/fk) as which 
the division of integrated value sigmaxk of the x-coordinate in each edge class k of a 
class table or each integrated value sigmayk of a y-coordinate is done in the frequency 
fk, and it is expressed in step S81 with the division value is searched for as a center of 
gravity of the pixel belonging to each edge class k. 

[0099] and the step S82 — progressing — the edge class 1 thru/or K -- it is alike, 
respectively, and the center of gravity of the center of gravity of the pixel which belongs 
is searched for as a center of gravity of the whole pixel which constitutes an attention 
frame, and carries out a return. 

[OlOO] namely, - step S82 -- the edge class 1 thru/or K " it is aUke, respectively, and the 
weighting average which makes weight the frequency fl thru/or fK is calculated about 
the center of gravity of the pixel which belongs, and it is outputted as a center of gravity 
of the whole pixel where the weighting average constitutes an attention frame. 
[OlOl] When searching for the center of gravity of an attention frame here only using 
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what belongs to an edge class among the pixels which constitute an attention frame, it 
is desirable to also search for the center of gravity of the motion detection range only 
using what belongs to an edge class among the pixels which constitute the motion 
detection range. 

[0102] In addition, in searching for the center of gravity of the motion detection range 
only using the pixel belonging to an edge class, in the processing shown in the flow chart 
of drawing 13 , it changes to processing of the step S71, and since it is the same as that 
of the case where processed steps S31 and S32 shown in the flow chart of drawing 10 , 
and also it is shown in drawing 13 , it omits the explanation. 

[0103] Next, when an attention frame is what does not have a camera motion to the 
frame in front of one of them, the camera motion vector of an attention frame should 
become equal to the camera motion vector of the frame in front of one of them. However, 
as it mentioned above, when asking for the camera motion vector of a frame, when a 
motion of a foreground influences, even if there is no camera motion of an attention 
frame, a different thing from the camera motion vector of the frame in front of one of 
them may be found as the camera motion vector. 

[0104] so, to the camera motion detecting element 12 shown in drawing 5 Form the 
motion-less judgment section 25 which judges whether an attention frame is what does 
not have a camera motion to the frame in front of one of them as a dotted line shows to 
this drawing, and it sets to the vector detecting element 22 further. In being a thing 
without a camera motion of an attention frame Output as a camera motion vector of an 
attention frame, and when an attention frame is a thing with a camera motion, the 
camera motion vector of the frame in front of one of them As it mentioned above, it can 
ask for the camera motion vector of an attention frame from the center of gravity of an 
attention frame and the motion detection range. 

[0105] By doing in this way, when it is a thing without a camera motion of an attention 
frame, the camera motion vector of an attention frame can be made equal to the camera 
motion vector of the frame in front of one of them. 

[0106] Then, with reference to the flow chart of drawing 15 , the processing (motion-less 
judgment processing) which judges whether it is the thing without a camera motion of 
an attention frame performed in the motion-less judgment section 25 is explained. 
[0107] In motion-less judgment processing, step S71 of drawing 13 thru/or the 
respectively same processing as the case in S79 are performed in step S91 thru/or S99. 
[0108] And in step S99, when judged with an attention pixel being an edge pixel, it 
progresses to step SlOO and it is judged whether it is equal to pixel value p' (x y) of the 
pixel which has the pixel value p of an attention pixel (x y) in the same location in front 
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of one of them. 

[0109] Here, p (x y) in step SlOO shall include the case where p (x y) is [ | p(x y)-p'(x y) | ] 
less than a predetermined minute value when almost equal to p' (x y) as it is equal to p' 

(xy). 

[OllO] In step SlOO, when judged with it not being equal to pixel value p' (x y) of the 
pixel which has the pixel value p of an attention pixel (x y) in the same location in front 
of one of them, step SlOl is skipped and it returns to step S96. 

[Olll] In step SlOO moreover, the pixel value p of an attention pixel (x y) When judged 
with it being equal to pixel value p' (x y) of the pixel in the same location in front of one 
of them, That is, when equal to pixel value p' (x y) of the pixel before I one ] the pixel 
value p of the attention pixel which is an edge pixel (x y) is in the same location spatially, 
it progresses to step SlOl, and the increment only of 1 is carried out and Variable c 
returns to step S96. 

[0112] Here, Variable c is cleared by zero before motion-less judgment processing of 
drawing 15 is started. 

[0113] Then, in step S94, when judged with Variable y not being under Y (i.e., when it 
processes by making into an attention pixel all the pixels that constitute an attention 
frame), it progresses to step S102 and it is judged whether Variable c is beyond the 
predetermined threshold th. When judged with Variable c being beyond the 
predetermined threshold th in step S102, Namely, the pixel used as a pixel value [ being 
the same as that of the pixel which is in the edge pixel of an attention frame in the same 
location of one frame ago (almost the same) ] When it exists beyond the threshold th, it 
progresses to step SlOS, and as a judgment result of whether an attention frame is what 
does not have a camera motion to the frame in front of one of them, the message of a 
purport without a motion is outputted to the vector detecting element 22, and ends 
motion-less judgment processing. 

[0114] Moreover, when judged with Variable c not being beyond the predetermined 
threshold th in step S102, Namely, the pixel used as a pixel value [ being the same as 
that of the pixel which is in the edge pixel of an attention frame in the same location of 
one frame ago (almost the same) ] When it does not exist beyond the threshold th, 
progress to step S104 and an attention frame receives the frame in front of one of them. 
As a judgment result of whether it is a thing without a camera motion, the message of a 
purport with a motion is outputted to the vector detecting element 22, and ends 
motion-less judgment processing. 

[0115] In addition, motion-less judgment processing of drawing 15 is performed in 
advance of the processing which computes the center of gravity of the attention frame in 
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the center-of-gravity calculation section 21, and the motion detection range, and is 
further performed for the frame after the 2nd frame. 

[0116] Next, drawing 16 shows the example of a configuration of the background extract 

section 14 of drawing 2 . 

[0117] The camera motion vector vl of the 1st frame thru/or the Nth frame as a series of 
images memorized by the camera motion vector storage section 13 ( drawing 2 ) thru/or 
vN are supphed to the existence range detecting element 31. The existence range 
detecting element 31 is in the condition which performed aUgnment of the background 
of the 1st thru/or the Nth frame, and detects the field (existence range) of the minimum 
rectangle where the 1st thru/or image of the Nth frame exists in standard coordinates. 
[0118] That is, based on the camera motion vector vl of the 1st frame thru/or the Nth 
frame thru/or vN, the existence range detecting element 31 is in the condition which 
performed the alignment about the 1st frame thru/or the Nth frame supposing the 
condition of having performed alignment of the background, and detects the existence 
range which is the field of the minimum rectangle where the 1st thru/or the pixel of the 
Nth frame exist. Furthermore, also in the standard coordinates of the existence range, 
the coordinate (Xmin, Ymin) of the top-most vertices of most the upper left and the 
coordinate (Xmax, Ymax) of the top-most vertices of most the lower right are searched 
for, and the existence range detecting element 31 is supplied to the read-out section 32 
and the write-in section 35. 

[0119] In addition, the condition of having performed ahgnment of the background of 
the Ist frame thru/or the Nth frame can be assumed in standard coordinates by 
arranging the 1st frame thru/or the Nth frame so that the top-most vertices at the upper 
left of a frame may be located in the coordinate shown by each camera motion vector vl 
thru/or vN, as shown in drawing 17 . 

[0120] The read-out section 32 detects and reads the pixel which is in the same location 
spatially, where alignment of the background of the 1st frame thria/or the Nth frame is 
performed among the pixels which constitute the 1st frame memorized by the are 
recording section 11 ( drawing 2 ) thru/or the Nth frame, and it supplies it to the 
frequency count area 33 and the write-in section 35. 

[012l] namely, -- the read-out section 32 — the coordinate (Xmin, Ymin) from the 
existence range detecting element 21 - and (Xmax, Ymax) it being supplied and also 
The camera motion vector vl memorized by the camera motion vector storage section 13 
( drawing 2 ) thru/or vN are also supplied. The read-out section 32 First, based on the 
camera motion vector vl of the 1st frame thru/or the Nth frame thru/or vN, the 
condition of having performed alignment of the background about the 1st frame thru/or 
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the Nth frame is assumed hke the existence range detecting element 31. Furthermore, 
the read-out section 32 carries out the sequential scan of the coordinate (Xmin, Ymin) 
supplied from the existence range detecting element 21 of standard coordinates, and 
(Xmax, Ymax) the coordinate of existence within the limits specified, and as shown in 
drawing 18 , it detects and reads the pixel of the 1st frame thru/or the Nth frame in the 
condition of having performed alignment of a background in each coordinate (x y). 
[0122] In addition, detection of the pixel of the n-th frame located in the coordinate (x y) 
of standard coordinates in the condition of having performed alignment of the 
background of the 1st thru/or the Nth frame When the local system of coordinates 
(suitably henceforth local system of coordinates) which made the zero the pixel at the 
upper left of [ the ] the n-th frame are considered so to speak, it can carry out by 
detecting the pixel which is in the coordinate which subtracted the camera motion 
vector vn from a coordinate (x y). 

[0123] The inside of the pixel to which the frequency count area 33 is supplied from the 
read-out section 32 and which constitutes the 1st thru/or the Nth frame, The pixel 
which is in the same location spatially where alignment of the background is performed 
every [ of (calling it hereafter the same location pixel in an aHgnment condition 
suitably) ] set - for example, the frequency of the pixel belonging to each level of the 
pixel value mentioned above -- counting — carrying out — the counting " based on a 
result, registration to the frequency table memorized by the frequency table storage 
section 34 of the latter part is performed. 

[0124] The frequency table storage section 34 memorizes a frequency table as shown in 
drawing 19 . That is, the frequency table storage section 34 has memorized the 
frequency table for matching and registering each level of the pixel value of the same 
location pixel in an alignment condition, and the rate to the number of pixels of the 
same location pixel in the alignment condition of the frequency of the pixel belonging to 
the level about each coordinate (x y) of existence within the limits. 

[0125] Here, in the set of the same location pixel in the alignment condition in a certain 
location (coordinate) of existence within the hmits, the level of the large pixel value to 
the m-th of frequency is called m-th frequency level. 

[0126] With the gestalt of operation of drawing 19 , M level and the rate of frequency 
from the 1st frequency level to the Mth frequency level are registered into a frequency 
table, therefore - here - the frequency count area 33 " frequency - the level 1st after 
M+-- counting of frequency -- a result is not registered into a frequency table but is 
canceled, however, counting of frequency " a result can also be made to register with a 
frequency table about all level 
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[0127] In addition, as mentioned above, when the range of the value which can be taken 
as a pixel value is divided into K level, above-mentioned M becomes a value below K. 
[0128] The write-in section 35 writes the background pixel which constitutes a whole 
background based on the frequency table memorized by the frequency table storage 
section 34 and the pixel supplied from the read-out section 32 in each address 
equivalent to existence within the limits supplied from the existence range detecting 
element 31 of the background memory 15 ( drawing 2 ). Furthermore, the write-in 
section 35 also performs the writing of the background flag to the background flag 
memory 36. 

[0129] The background flag memory 36 memorizes the background flag with which it 
expresses whether the background pixel is written in about each pixel of existence 
within the limits. That is, the write-in section 35 writes a background flag in the 
address of the background flag memory 36 corresponding to the address, when a 
background pixel is written in the address with the background memory 15. Here, a 
background flag shall presuppose that it is a 1-bit flag, the background flag 
corresponding to the address with which the background pixel is written in shall be set 
to 1, and the background flag corresponding to the address which is not written in yet 
shall be set to 0. 

[OlSO] Next, with reference to the flow chart of drawing 20 , background extract 
processing in which a whole background is extracted is explained from the 1st frame 
performed in the background extract section 14 of drawing 16 thru/or the Nth frame. 
[0131] First, in step Sill, as the existence range detecting element 31 reads a camera 
motion vector from the camera motion vector storage section 13 and explained it by 
drawing 17 , it detects the existence range. And the coordinate (Xmin, Ymin) of the 
point of the upper left in the standard coordinates of the existence range and the 
coordinate (Xmax, Ymax) of a lower right point are supplied to the read-out section 32 
and the write-in section 35 as information for pinpointing the existence range. 
[0132] Ymin-1 as initial value is set to the coordinate (Xmin, Ymin) for pinpointing the 
existence range, and (Xmax, Ymax) the variable y for scanning the existence range to y 
shaft orientations of standard coordinates in step Si 12, if it receives, it progresses to 
step SI 13, only 1 increments the variable y, and the read-out section 31 progresses to 
step Si 14. At step Si 14, when judged with it being judged whether Variable y is below 
Ymax and being below Ymax, it progresses to step S115. At step S115, the read-out 
section 31 sets Xmin-1 as initial value to the variable x for scanning the existence range 
in the direction of a x axis of standard coordinates, progresses to step Si 16, and only 1 
increments the variable x and it progresses to step S117. At step S117, when judged 
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with it being judged whether Variable x is below Xmax and not being below Xmax, the 
same processing is repeated by step Si 13 return and the following. 
[0133] Moreover, in step S117, when judged with Variable x being below Xmax, it 
progresses to step Si 18, and 0 as initial value is set to the variable n for counting the 
frame number of the 1st frame thrti/or the Nth frame as a series of images memorized 
by the are recording section 11 ( drawing 2 ), and it progresses to step Si 19. At step 
S119, it is judged whether Variable n is below N that is the frame number of a series of 
images with which the increment only of 1 was carried out, it progressed to step S120, 
and Variable n was memorized by the are recording section 11 ( drawing 2 ). 
[0134] In step S120, when judged with Variable n being below N, it progresses to step 
S121, and where aUgnment of the background of the 1st frame thru/or the Nth frame is 
performed, in the read-out section 32, reading appearance of the pixel of the n-th frame 
in the location of the coordinate (x y) of standard coordinates is carried out from the are 
recording section 11 ( drawing 2 ). That is, the read-out section 32 is the coordinate (x y) 
which subtracted the camera motion vector vn from the coordinate (x y) in the local 
system of coordinates which made the zero the pixel at the upper left of the n-th frame. ■ 
The pixel in vn is read from the are recording section 11 ( drawing 2 ). 
[0135] In addition, while expressing the x-coordinate or ycoordinate of the camera 
motion vector vn as xv#n or yv#n, respectively If the side of one frame or the vertical 
number of pixels is expressing X or Y, respectively, although the read-out section 32 will 
read the pixel located in the coordinate (x-xv#n, y-yv#n) in the local system of 
coordinates about the n-th frame In this case, the pixel of the n-th frame does not exist 
in the coordinate (x-xv#n, y-yv#n) of 0 <=x-xv#n<X and 0 <=y-jrv#n<Y which can be set 
out of range. Therefore, read-out of the pixel of the n-th frame in step S121 is performed, 
only when x-xv#n is within the limits of 0 <=x-xv#n<X and y-yv#n is within the limits of 
0 <=y-yv#n<Y. 

[0136] In step S121, if the pixel of the n-th frame is read, the read-out section 32 will 
supply the pixel to the frequency count area 33 and the write-in section 35, and will 
return from the are recording section 11 ( drawing 2 ) to step Si 19. And until it is 
judged with Variable n not being below N in step S120 Step S119 thru/or processing of 
S121 are repeated, and by this, where alignment of the background of the 1st frame 
thru/or the Nth frame is performed The pixel (the same location pixel in an alignment 
condition) of the 1st frame in the location of the coordinate (x y) of standard coordinates 
thru/or the Nth frame is supplied to the frequency count area 33 and the write-in 
section 35. However, as mentioned above, depending on a frame, a pixel may not exist in 
the coordinate (x y) of standard coordinates, and the pixel of that frame is not contained 
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in the pixel supplied to the frequency count area 33 and the write-in section 35 in this 

case. 

[0137] If judged with Variable n not being below N in step S120, it will progress to step 
S122. Then, the frequency count area 33 The pixel value carries out the level 
classification of each pixel of the 1st thru/or the Nth frame in the condition of having 
performed alignment of a background in the location of the coordinate (x y) of standard 
coordinates supplied from the read-out section 32 by whether it belongs to level 1 
thru/or which range of the K. Furthermore, the frequency count area 33 carries out 
counting of the frequency of the pixel belonging to each level, and asks for the rate (rate 
to the total of the pixel belonging to each level) of the frequency. 

[0138] And it progresses to step S123, and the frequency count area 33 is registered into 
the column (hne) of a coordinate (x y) in a frequency table as showed the rate of the 
frequency of a pixel that the rate of frequency belongs to the level (the 1st frequency 
level thru/or the Mth frequency level) and each of its level from the 1st place to the Mth 
place to drawing 19 of the frequency table storage section 34, and returns to step S116. 
[0139] When it is judged with Variable y not being below Ymax in step SI 14 on the 
other hand, When all the coordinates of existence within the limits are processed, it 
progresses to step Si 24. Namely, the write-in section 35 The coordinate which the rate 
of the pixel belonging to the 1st frequency level has become in the frequency table 
memorized by the frequency table storage section 34 beyond the predetermined value is 
detected. Background pixel extract processing which writes in the pixel value 
corresponding to the 1st frequency level as a pixel value of a background pixel is 
performed to the address of the background memory 15 corresponding to the coordinate, 
and it goes to it at step S125. At step S125, about the coordinate of existence within the 
limits in which a pixel value was not written in background pixel extract processing of 
step S124, the write-in section 35 performs background escape processing which writes 
in the pixel value as a background pixel, and ends background extract processing. 
[0140] Next, with reference to the flow chart of drawing 21 , the background pixel 
extract processing which writes in in step 8124 of drawing 20 , and the section 35 
performs is explained. 

[0141] In background pixel extract processing, in step S131 thru/or S136, step S112 of 
drawing 20 thru/or the respectively same processing as the case in 117 are performed, 
and it sets to step S136. When judged with Variable x being below Xmax, it is judged by 
progressing to step S37 and referring to a frequency table whether the rate of the 1st 
frequency level to a coordinate (x y) is beyond the predetermined threshold Lth. 
[0142] When judged with the rate of the 1st frequency level to a coordinate (x y) not 
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being beyond the predetermined threshold Lth in step S137, That is, when the rate of 
the frequency of the pixel value with most frequency of the pixel located in a coordinate 
(x y) among the pixels which constitute the 1st frame in the condition of having 
performed alignment of a background thru/or the Nth frame is not high, steps S138 and 
S139 are skipped, and it returns to step S135. 

[0143] Therefore, the pixel value of a background pixel is not written in the address of 
the background memory 15 ( drawing 2 ) corresponding to the coordinate (x y) in the 
existence range in this case. 

[0144] When it is judged with the rate of the 1st frequency level to a coordinate (x y) 
being beyond the predetermined threshold Lth in step S 137 on the other hand. Namely, 
the inside of the pixel which constitutes the 1st frame in the condition of having 
performed alignment of a background thru/or the Nth frame, When the rate of the 
frequency of the pixel value with most frequency of the pixel located in a coordinate (x y) 
is high, it progresses to step SI 38 and the write-in section 35 writes a pixel value with 
the high rate of the frequency in the background memory 15 as a pixel value of the 
background pixel located in a coordinate (x y). 

[0145] That is, the write-in section 35 extracts what belongs to the 1st frequency level 
among the pixels located in the coordinate (x y) of the 1st frame in the condition of 
having performed aUgnment of a background thru/or the Nth frame from the pixel 
supplied from the read-out section 32, for example, calculates the average. And the 
write-in section 35 writes the average in the background memory 15 as a pixel value of 
the background pixel located in a coordinate (x y). 

[0146] After processing of step S138, it progresses to step S139, the write-in section 35 
sets to 1 the background flag memorized to the address corresponding to the coordinate 
(x y) of the background flag memory 36 (leave a background flag), and the same 
processing is repeated until it is judged with Variable y not being below Ymax by step 
S135 in step S133 return and the following. 

[0147] And in step S133, if judged with Variable y not being below Ymax, a return will 
be carried out. 

[0148] Here, the storage value of the background flag memory 36 is cleared by 0 when 
background pixel extract processing of drawing 21 is started. 

[0149] Next, in background pixel extract processing in which it explained by drawing 
21 , as mentioned above, when the rate of the frequency of the pixel value with most 
frequency of the pixel located in a coordinate (x y) among the pixels which constitute the 
1st frame in the condition of having performed alignment of a background thru/or the 
Nth frame is not high, the pixel value as a background is not written in a coordinate (x 



JP11258473 



y). 

[0150] Namely, the inside of the pixel which constitutes the 1st frame in the condition of 
having performed alignment of a background thru/or the Nth frame from background 
pixel extract processing, When the rate of the frequency of the pixel value with most 
frequency of the pixel located in a coordinate (x y) is high, it is written in the 
background memory 15 as a thing with most pixel values of the frequency probable as a 
pixel value of the whole background in a coordinate (x y). therefore, when most pixel 
values of frequency cannot say that it is probable as a pixel value of the whole 
background in a coordinate (x y), here The inside of the pixel which constitutes the 1st 
frame in the condition of having performed alignment of a background thru/or the Nth 
frame, When the rate of the frequency of the pixel value with most frequency of the 
pixel located in a coordinate (x y) is not high, in background pixel extract processing, the 
pixel value of a whole background is not written in the address of the background 
memory 15 corresponding to a coordinate (x y). 

[0151] Consequently, by background pixel extract processing, so to speak, the whole 
background which consists of that a pixel value is written in the background memory 15 
is in the vermin condition, and needs to bury the vermin part. For this reason, in 
background extract processing of drawing 20 , after background pixel extract processing 
( drawing 21 ) is performed at step S124, in step S125, background escape processing 
which extends the background so to speak is performed by fill uping a vermin part with 
a pixel value. 

[0152] Then, with reference to the flow chart of drawing 22 , the background escape 
processing in step S125 of drawing 20 is explained. 

[0153] In background escape processing, in step S141, Ymin-1 as initial value is set to 
Variable y, and it progresses to step S142, and the increment only of 1 is carried out and 
the variable y progresses to step S143. At step S143, when judged with it being judged 
whether Variable y is below Ymax and being below Ymax, it progresses to step S145. At 
step S145, Xmin-1 as initial value is set to Variable x, and it progresses to step S146, 
and the increment only of 1 is carried out and the variable x progresses to step S147. At 
step S147, when judged with it being judged whether Variable x is below Xmax and not 
being below Xmax, the same processing is repeated by step S142 return and the 
following. 

[0154] Moreover, in step S147, when judged with Variable x being below Xmax, it 
progresses to step S148 and it is judged whether the background flag memorized to the 
address of the background flag memory 36 corresponding to a coordinate (x y) is 0. 
When judged with the background flag memorized to the address of the background flag 
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memory 36 corresponding to a coordinate (x y) not being 0 in step S148, That is, the 
background flag is 1, therefore when the pixel value as a whole background is already 
written in the address of the background memory 15 ( drawing 2 ) corresponding to a 
coordinate (x y), the same processing is repeated by step S146 return and the following. 
[0155] Moreover, when judged with the background flag memorized to the address of 
the background flag memory 36 corresponding to a coordinate (x y) being 0 in step S148, 
To namely, the address of the background memory 15 ( drawing 2 ) corresponding to a 
coordinate (x y) When the pixel value as a whole background is not written in, it still 
progresses to step S149. Moreover as a surrounding pixel of the pixel located in a 
coordinate (x y), the bottom, it is judged because whether the background flag about 
either of the pixels which adjoin the left, the right, the upper left, the lower left, the 
upper right, and the lower right is 1 refers to the background flag memory 36. 
[0156] When judged with neither of the background flags of the pixel which adjoins the 
pixel located in a coordinate (x y) in step S149 being 1, Namely, a coordinate (x yl), (x, 
y+l), (x-1, y), (x+1, y), When the pixel value as a whole background is written in neither 
of the address of the background memory 15 ( drawing 2 ) corresponding to (x-1, yl), 
(x-1, y+l), (x+1, y-l), and each (x+1, y+l) yet, it returns to step S146. 
[0157] Moreover, when judged with one background flag of the pixels which adjoin the 
pixel located in a coordinate (x y) in step S149 being 1, Namely, a coordinate (x y-l), (x, 
y+l), (x-1, y), (x+1, y). When the pixel value as a whole background is already written in 
either of the addresses of the background memory 15 ( drawing 2 ) corresponding to (x-1, 
y-l), (x-1, y+l), (x+1, y-l), and each (x+1, y+l), it progresses to step S150. 
[0158] Here, that in which the pixel value as a whole background is written among the 
pixels (address of the corresponding background memory 15 ( drawing 2 )) which adjoin 
the pixel located in a coordinate (x y) is hereafter called written in contigmty pixel 
suitably. 

[0159] At step Si 50, it is judged whether the level of the pixel value of a written in 
contiguity pixel and a continuous pixel value is registered into the frequency table as 
either the 1st to a coordinate (x y) or thru/or the Mth frequency level. Here, a continuous 
pixel value means pixel values (the same pixel value is included) with a near value. 
[0160] In step S150, the level of the pixel value of a written in contiguity pixel and a 
continuous pixel value as either the 1st to a coordinate (x y) or thru/or the Mth 
frequency level When judged with registering with the frequency table, constitute the 
1st frame in the condition of having performed alignment of a background thrWor the 
Nth frame. What has the pixel value which follows the pixel value of a written in 
contiguity pixel in the pixel located in a coordinate (x y) the case where it exists more 
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than a certain amount of rate -- step S151 - progressing -- the -- Based on the pixel 
value which exists more than a certain amount of rate and which follows the pixel value 
of a written in contiguity pixel, the pixel value as a whole pixel background located in a 
coordinate (x y) is computed, and it is written in the address with which the background 
memory 15 ( drawing 2 ) corresponds. 

[0161] Namely, supposing the level of the pixel value of a written in contiguity pixel and 
a continuous pixel value was registered into the frequency table as the m-th frequency 
level of the Ist to a coordinate (x y) thru/or the Mth frequency level Constitute the 1st 
frame in the condition of having performed alignment of a background thru/or the Nth 
frame from a step S151. The average of the pixel value belonging to the m-th frequency 
level of the pixels located in a coordinate (x y) is computed, and the average is written in 
the address with which the background memory 15 ( drawing 2 ) corresponds as a pixel 
value as a whole pixel background located in a coordinate (x y). 

[0162] As mentioned above, here in background pixel extract processing The inside of 
the pixel which constitutes the 1st frame in the condition of having performed 
aUgnment of a background thru/or the Nth frame. Although it is written in the 
background memory 15 as a thing with most pixel values of the frequency probable as a 
pixel value of the whole background in a coordinate (x y) when the rate of the frequency 
of the pixel value with most frequency located in a coordinate (x y) is high For example, 
a certain scenery is made into a background, according to this background pixel extract 
processing When a series of images which become with the N frame which a certain 
body moves as a foreground are considered Since the rate of the frequency of the pixel 
value currently continued and displayed on almost all the frames about the pixel as 
which most N frames are covered and the background is displayed becomes high, 
without being hidden by the foreground, the pixel value is written in as a pixel value of 
a background. 

[0163] However, the pixel (suitably henceforth a medium pixel) as which a foreground is 
displayed when a foreground moves, or a background is displayed Since it becomes the 
pixel value which constitutes a foreground or becomes the pixel value which constitutes 
a background, since the pixel value to which the rate of frequency becomes high does not 
exist, about a medium pixel, the writing of a pixel value is not performed by backgroiind 
pixel extract processing. 

[0164] Although what is necessary is just to write in the pixel value of the frame as 
which the background is displayed there about a medium pixel in order to acquire a 
whole background, it is difficult to specify the frame as which the background is 
displayed on the medium pixel. Then, if there are some in which the pixel value as a 
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whole background is already written and the pixel value and a near pixel value may be 
displayed as a pixel value of a medium pixel in background escape processing into the 
pixel which adjoins a medium pixel It is written in the background memory 15 
( drawing 2 ) noting that a pixel value with the thing [ having been displayed ] is 
probable as a pixel value when a background is displayed on a medium pixel. 
[0165] Therefore, according to background escape processing, it becomes possible to 
write in the pixel value of the background extremely displayed in one of them only in 
one certain frame also about the pixel (medium pixel) as which the background was 
displayed. 

[0166] On the other hand in step S150, the level of the pixel value of a written in 
contiguity pixel and a continuous pixel value as either the 1st to a coordinate (x y) or 
thru/or the Mth frequency level When judged with not registering with a frequency 
table, constitute the 1st frame in the condition of having performed alignment of a 
background thrWor the Nth frame. When what has the pixel value which follows the 
pixel value of a written in contiguity pixel does not exist in the pixel located in a 
coordinate (x y), It progresses to step S152, the pixel value as a whole pixel background 
located in a coordinate (x y) is computed based on the pixel value of a written in 
contiguity pixel, and it is written in the address with which the background memory 15 
( drawing 2 ) corresponds. 

[0167] Namely, although some (written in contiguity pixel) in which the pixel value as a 
whole background is already written are in the pixel which adjoins a medium pixel 
When the pixel value and a near pixel value are not displayed as a pixel value of a 
medium pixel In step S152, the pixel value (for example, the average when two or more 
written in contiguity pixels exist) of a written in contiguity pixel is written in the 
background memory 15 ( drawing 2 ) as a pixel value as a whole pixel background 
located in a coordinate (x y). 

[0168] After processing of steps S151 and S152, it progresses to step S153, the 
background flag of the address corresponding to the coordinate (x y) of the background 
flag 36 of a pixel with which the pixel value as a whole background was written in at 
steps S151 or S152, i.e., background flag memory, is set to 1 by each (built), and the 
same processing is repeated by step S146 return and the following. 
[0169] On the other hand, when judged with Variable y not being below Ymax in step 
SI 43, it is judged whether all the background flags to each coordinate of existence 
within the limits which progressed to step S144 and was memorized by the foreground 
flag memory 36 are 1. In step SI 43, when judged with some which are not 1 being in the 
background flag corresponding to each coordinate of existence within the limits (i.e., 
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when some in which the pixel value as a whole hackground is not written yet are in the 
pixel located in the coordinate of existence within the limits), the same processing is 
repeated by step S141 return and the following. 

[0170] Moreover, in step S144, when judged with there being nothing that is not 1 into 
the background flag corresponding to each coordinate of existence within the limits (i.e., 
when the pixel value as a whole background is written in all the pixels located at the 
coordinate of existence within the limits), a return is carried out. 

[0171] The pixel in which according to the processing explained by drawiner 22 the pixel 
value as a whole background was already written as shown in drawing 23 (in this 
drawing) - When the pixel P (x y) (O mark which attached the slash shows this drawing) 
which adjoins that the mark shows and in which the pixel value is not written yet exists 
The written in contiguity pixel which adjoins the pixel P (x y) (in this drawing) The pixel 
value of Pixels P (x-1, y), P (x-1, y-l), P (x yl), and P (x+1, y+l), the pixel value, the 
pixel value which has a continuity, etc. are written in as a pixel value as a whole pixel P 
(x y) background, and, thereby, the whole backgrovmd is extended. From this, 
processing of drawing 22 is called background escape processing. 

[0172] Next, drawing 24 shows the example of a configuration of the foreground coding 

section 16 of drawing 2 . 

[0173] The 1st frame as a series of images memorized by the are recording section 11 
( drawing 2 ) thru/or the Nth frame, the whole background memorized by the 
background memory 15 ( drawing 2 ), and the camera motion vector memorized by the 
camera motion vector storage section 13 ( dx'awing 2 ) are supplied to the foreground 
extract section 41, and the foreground extract section 41 extracts a foreground from the 
1st frame thru/or each Nth frame. That is, it is arranging the n-th frame, and the 
foreground extract section 41 performs alignment of a whole background and the n-th 
frame, it is subtracting the pixel of the whole background in the same location from each 
pixel of the n-th frame, and extracts a foreground from the n-th frame so that the point 
at the upper left of the n-th frame may be located in the location where only the camera 
motion vector vn shifted in the standard coordinates of a whole background. 
[0174] The foreground storage section 42 memorizes the 1st thru/or the foreground of 
the Nth frame which the foreground extract section 41 extracted. 
[0175] The camera motion vector memorized by the camera motion vector storage 
section 13 ( drawing 2 ) is supplied to the foreground are recording image configuration 
section 43, and the foreground are recording image configuration section 43 constitutes 
a front are recording image and a back are recording image using the foreground of the 
1st frame thru/or the Nth frame memorized by the foregrovmd storage section 42 based 
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on the camera motion vector. Namely, the foreground are recording image configuration 
section 43 is based on a camera motion vector, as shown in drawing 25 . The foreground 
of the 1st frame thru/or the Nth frame in the condition of having performed ahgnment 
of a background is assumed. The back are recording image obtained when the 
foreground of the 1st frame thrWor the Nth frame is seen toward the travelling 
direction of time amount (image constituted by the foreground at which it looked from 
the past), The front are recording image (image constituted by the foreground at which 
it looked from the future) obtained when it goes to the traveUing direction and hard flow 
of time amount is constituted. 

[0176] In addition, a front are recording image is the sequence of the 1st frame to the 
Nth frame, and can acquire the pixel value of the foreground of the 1st frame thru/or the 
Nth frame in the condition of having performed ahgnment of a background, by 
overwriting memory etc. Similarly, a back are recording image is the sequence of the 
Nth frame to the 1st frame, and can acquire the pixel value of the foreground of the Ist 
frame thru/or the Nth frame in the condition of having performed alignment of a 
background, by overwriting memory etc. 

[0177] The foreground are recording image storage section 44 memorizes the front are 
recording image and back are recording image which were constituted in the foreground 
are recording image configuration section 43. 

[0178] The study section 45 performs study processing which asks for the prediction 
coefficient for predicting the pixel which constitutes the foreground of the 1st frame 
thru/or each Nth frame firom the pixel which constitutes the front are recording image 
memorized by the foreground are recording image storage section 44 and a back are 
recording image, and the error image which was further memorized by the error image 
storage section 49 if needed, and which is mentioned later. 

[0179] The prediction coefficient storage section 46 memorizes the prediction coefficient 
called for by performing study in the study section 45. 

[0180] The adaptation processing section 47 performs adaptation processing which 

predicts the foreground of the 1st frame thru/or each Nth frame using the pixel which 

constitutes the front are recording image and back are recording image which were 

memorized by the foreground are recording image storage section 44, the prediction 

coefficient memorized by the prediction coefficient storage section 46, and the error 

image further memorized by the error image storage section 49 if needed. 

[0181] Here, the study processing performed in the study section 45 and the adaptation 

processing performed in the adaptation processing section 47 are explained. 

[0182] The forecast of the pixel which constitutes the image (here a front are recording 



JP11258473 



image and a back are recording image, and an error image) which exists now from 
adaptation processing, for example, and the pixel which constitutes a desired image 
(here foreground of the 1st frame thru/or each Nth frame) by linear combination with a 
predetermined prediction coefficient is calcvilated. 

[0183] On the other hand, by study processing, while using a desired image (suitably 
henceforth a request image) as educator data The image (suitably henceforth an 
existence image) which exists when it is going to ask for the request image is used as 
student data. Forecast [ of the pixel value y of the pixel (suitably henceforth a request 
pixel) which constitutes a request image ] E [y] For example, the pixel value xl of a 
some existence pixel (pixel which constitutes an existence image), x2, and the set of..., 
The prediction coefficient for asking with the primary linearity joint model specified by 
the predetermined prediction coefficients wl and w2 and the linear combination of ... is 
called for. In this case, forecast E ly] can be expressed with a degree type. 
[0184] 

E[y] =wlxl+w2 x2+ (l) 

[0185] It is [Equation l] about matrix Y' which becomes by the matrix X which becomes 
by the matrix W which becomes by the set of a prediction coefficient wj, and the student 
data aggregate in order to generalize a formvila (l), and the set of forecast E [y]. 
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If a definition is come out and given, the following observation equations will be 

materialized. 

[0186] 

XW=Y' ... (2) 

Here, the component xij of Matrix X means the j-th student data in the student data 
aggregate (student data aggregate used for prediction of the educator data yi of the i-th 
affair) of the i-th affair, and the component wj of Matrix W expresses the prediction 
coefficient which a product with the j-th student data in the student data aggregate 
calculates. Moreover, yi expresses the educator data of the i-th affair, therefore E [yi] 
expresses the forecast of the educator data of the i-th affair. In addition, y in the left 
part of a formula (l) omits the suffix i of the component yi of Matrix Y, and xl in the 
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right-hand side of a formula (l), x2, and ... also omit the suffix i of the component xij of 
Matrix X. 

[0187] And it considers applying a least square method to this observation equation, 
and asking for forecast E [y] near the pixel value y of a request pixel. In this case, it is 
[Equation 2] about the matrix E which becomes by the set of the matrix Y which 
becomes by the set of the true pixel value y of the request pixel used as educator data, 
and the remainder e of forecast E [y] to the pixel value y of a request pixel. 
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If a definition is come out and given, the following remainder equations will be 

materialized from an equation (2). 

[0188] 

XW=Y-i-E ... (3) 

[0189] In this case, the prediction coefficient wj for asking for forecast E [y] near the 
pixel value y of a request pixel iiS a square error [several 3]. 

It can ask by making it min. 

[0190] Therefore, it will be called an optimum value, when what differentiated the 
above-mentioned square error with the prediction coefficient wj is set to 0, namely, in 
order that the prediction coefficient wj which fills a degree type may ask for forecast E 
[y] near the pixel value y of a request pixel. 
[0191] 

[Equation 4] 

eil^L |e2.^ ML^O ( j=i.2.....J) 
3Wj " 9Wj awj 

... (4) 

[0192] Then, a degree type is first materialized by differentiating a formula (3) with a 

prediction coefficient wj. 

[0193] 

[Equation 5] 
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... (5) 

[0194] A formula (6) is obtained from a formiila (4) and (5). 
[0195] 

[Equation 6] 




... (6) 



[0196] Furthermore, if the relation of the student data xij in the remainder equation of 
an equation (3), a prediction coefficient wj, the educator data yi, and Remainder ei is 
taken into consideration, the following normal equations can be obtained from an 
equation (6). 
[0197] 

[Equation 7] 

' ( I XilXil)W,+ (|^ XilXi2)W2+... + (^ XilXij)Wj=( j: XilYi) 
( y Xi2Xii)l«m-(^ Xi2Xi2)W2+... + (y Xj2Xij)Wj=( J: Xi2yi) 



[0198] Each equation which constitutes the normal equation of an equation (7) is that 
only a certain amount of nvimber prepares the set of the student data xij and the 
educator data yi. Only the same number as several J of the prediction coefficient wj for 
which it should ask can be built, therefore a formvda (7) can be asked for the optimal 
prediction coefficient wj by solution Lycium chinense (however, in order to solve a 
formula (7), in a formula (7), the matrix which consists of multipliers concerning a 
prediction coefficient wj needs to be regular). In addition, in solving a formula (7), it is 
possible to sweep out and to, use law (method of elimination of Gauss-Jordan) etc. for 



[0199] Study processing asks for the optimal prediction coefflcient wj as mentioned 




+ (.1 XijXij)Wj: 




... (7) 



example. 
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above, and adaptation processing asks for forecast E [y] near the pixel value y of a 
request pixel by the formula (l) using the prediction coefficient wj. 
[0200] That is, in the study section 45 or the adaptation processing section 47, while 
using as an existence image the front are recording image memorized by the foreground 
are recording image storage section 44 and a back are recording image, and the error 
image further memorized by the error image storage section 49 if needed, study 
processing or adaptation processing is performed, respectively considering the 
foreground of the 1st frame thru/or each Nth frame as a request image. 
[0201] In addition, although adaptation processing is not included in an existence image, 
it is the point that the component contained in a request image is reproduced, for 
example, differs from mere interpolation processing. That is, in adaptation processing, 
as long as a formula (l) is seen, it is the same as that of the interpolation processing 
using the so-called interpolation filter, but since [ for which the prediction coefficient w 
equivalent to the tap multiplier of the interpolation filter uses the educator data y ] it 
asks by study so to speak, the component contained in a request image is reproducible. 
From this, adaptation processing can be called processing which, so to speak, has a 
creation (resolution imagination) operation of an image. 

[0202] The error count section 48 reads the foreground of the 1st frame thru/or each Nth 
frame from the foreground storage section 42, and calculates the prediction error of the 
forecast of the foreground of the 1st frame thru/or each Nth frame called for in the 
adaptation processing section 47. Namely, the error count section 48 calculates a 
prediction error for every pixel by subtracting the true value of the pixel value of the 
pixel from the forecast of the pixel which constitutes the foreground of the n-th frame. 
[0203] The error image storage section 49 memorizes the image (suitably henceforth an 
error image) which becomes by the prediction error of the foreground of the 1st frame 
thru/or the Nth frame called for in the error count section 48. 

[0204] The error judging section 50 calculates for example, the absolute value sum of 
the prediction error as a pixel value of each pixel which constitutes the error image 
memorized by the error image storage section 49, and judges whether the absolute 
value sum has become below a predetermined threshold (following). 
[0205] MUX (multiplexer)51 is based on the judgment result by the error judging 
section 50. To the prediction coefficient and pan which were memorized by the front are 
recording image memorized by the foreground are recording image storage section 44 
and a back are recording image, and the prediction coefficient storage section 46 The 
error image memorized by the error image storage section 49 is multiplexed if needed, 
and the multiplexing data obtained as a result are outputted to a multiplexer 17 
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( drawing 2 ) as a coding result of the foreground of the 1st &ame thru/or each Nth 
frame. 

[0206] Next, drawing 26 shows the example of a configuration of the study section 45 of 

drawing 24 . 

[0207] The front are recording image memorized by the foreground are recording image 
storage section 44 and a back are recording image (suitably henceforth [ both are 
included and ] a foreground are recording image), and the error image further 
memorized by the error image storage section 49 if needed are supphed to the prediction 
tap configuration section 61. And the prediction tap configuration section 61 sets as an 
attention pixel what is going to calculate a forecast among the pixels which constitute 
the foreground of the 1st frame thru/or each Nth frame in the condition of having 
performed alignment of a background to standard coordinates. The pixel of the front are 
recording image in the location near an attention pixel and a space target and a back 
are recording image and the pixel of an error image are extracted, and it outputs to the 
normal equation configuration section 62 as a prediction tap used for calculating the 
forecast of an attention pixel by adaptation processing. 

[0208] A prediction tap is supplied from the prediction tap configuration section 61, and 
also the pixel which constitutes the foreground of the 1st frame memorized by the 
foreground storage section 42 thru/or each Nth frame is supplied to the normal equation 
configuration section 62. And the normal-equation configviration section 62 performs 
the add lump for the pixel (educator data) and prediction tap (student data) of the 
foreground used as an attention pixel. 

[0209] That is, the normal equation configuration section 62 performs the multiplication 
(xijxi'jO of the student data (prediction tap) used as the multiplier of a prediction 
coefficient in the left part of the normal equation of an equation (7), and the operation 
equivalent to a summation (sigma) using a prediction tap. 

[0210] Furthermore, the normal equation configuration section 62 performs the 
multiphcation (xijyj) of the student data (prediction tap) in the right-hand side of the 
normal equation of an equation (7), and educator data (attention pixel), and the 
operation equivalent to a summation (sigma) using a prediction tap and an attention 
pixel. 

[02 ll] The normal equation having held as an attention pixel and having shown by this 
the pixel from which the above processing constitutes the foreground of the 1st frame 
thru/or each Nth frame at a ceremony (7) is built in the normal equation configuration 
section 62. 

[0212] And the prediction coefficient calculation section 63 asks for a prediction 
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coefficient, and makes the prediction coefficient storage section 46 ( drawing 24 ) supply 
and memorize after that the normal equation generated in the normal equation 
configuration section 62 by solution Lycium chinense. 

[0213] Here, with the gestalt of this operation, forecast [ of the pixel value An (x y) of the 
pixel which is in the location (x y) of n-th-frame standard coordinates' existence within 
the Umits among the 1st frame in the condition of having performed ahgnment of a 
background thru/or the Nth frame ] E [An (x y)] is called for in the adaptation 
processing section 47 of drawing 24 for example, according to a degree type. 
[0214] 

E[An(x,y)]=g(F,B,E,n) 
... (8) 

Here, in a formula (8), F, B, and E express a front are recording image, a back are 
recording image, and an error image, respectively, and Function g (F, B, E, n) is defined 
by the degree type equivalent to the linearity linear expression of a formula (l). 
[0215] 

g(F, B, E, n) =wFlxfl+wF2xf2+ ... + wBlxbl+wB2xb2+ ... + wElxel+wE2xe2+ ... 
+wxn ... (9) 

Here, in a formula (9), wFl, wF2, wBl and wB2, wEl, wE2 \y express a 

prediction coefficient. Moreover, the pixel which constitutes the prediction tap about an 
attention pixel among the pixels from which el, e2, and ... constitute the error image E 
for the pixel which constitutes the prediction tap about an attention pixel among the 
pixels from which bl, b2, and ... constitute the back are-recording image B for the pixel 
which constitutes the prediction tap about an attention pixel among the pixels from 
which fl, £2, and ... constitute the front are recording image F is expressed, respectively. 
[0216] When Function g (F, B, E, n) is defined by the equation (9), in the normal 
equation configvuration section 62 of drawing 26 The normal equation for asking for the 
prediction coefficients wFl and wF2 in an equation (9), .... wBl and wB2, wEl, 
wE2, w is built. In the prediction coefficient calculation section 63 Prediction 
coefficients wFl and wF2, wBl and wB2, wEl, wE2, w are asked for the 
normal equation by solution Lycium chinense. Therefore, the prediction coefficients 
wFl and wF2 of one set, wBl and wB2, wEl, wE2, w are called for about the 
foreground of the 1st frame thru/or all the Nth frame in this case. 
[0217] Next, drawing 27 shows the example of a configuration of the adaptation 
processing section 47 of drawing 24 . 

[0218] The foreground are recording image memorized by the foreground are recording 
image storage section 44 and the error image further memorized by the error image 
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storage section 49 if needed are supplied to the prediction tap configuration section 71. 
and the prediction tap configuration section 71 like the case in the prediction tap 
configuration section 61 of drawing 26 What is going to calculate a forecast among the 
pixels which constitute the foreground of the 1st frame thru/or each Nth frame in the 
condition of having performed alignment of a background is set to standard coordinates 
as an attention pixel. The pixel of the front are recording image in the location near an 
attention pixel and a space target and a back are recording image and the pixel of an 
error image are extracted, and it outputs to the prediction operation part 72 as a 
prediction tap. 

[0219] A prediction tap is supplied from the prediction tap configuration section 71, and 
also the prediction coefficient memorized by the prediction coefficient storage section 46 
( drawing 24 ) is supplied to the prediction operation part 72. And in the prediction 
operation part 72, the forecast of the pixel of a foreground used as an attention pixel is 
calculated by calcvdating the linearity linear expression defined by a formula (8) and (9) 
using a prediction tap and a prediction coefficient, and is outputted to the error count 
section 48. 

[0220] Next, with reference to the flow chart of drawing 28 , the foreground coding 
processing which encodes the foreground of the 1st frame thru/or each Nth frame 
performed in the foreground coding section 16 of drawing 24 is explained. 
[0221] It sets to step S161 first. The foreground extract section 41 The camera motion 
vector memorized by the camera motion vector storage section 13 ( drawing 2 ), And as 
it mentioned above, a foreground is extracted, and the foreground storage section 42 is 
supplied and is made to memorize using the whole background memorized by the 
background memory 15 ( drawing 2 ) from each image of the 1st frame thru/or the Nth 
frame memorized by the are recording section 11 ( drawing 2 ). 
[0222] And it progresses to step S162, and the foreground are recording image 
configuration section 43 constitutes a front are recording image and a back are 
recording image which were explained by drawing 25 from a foreground of the 1st frame 
memorized by the foreground storage section 42 thru/or each Nth frame, and the 
foreground are recording image storage section 44 is made to supply and memorize it, 
and it progresses to step 163. 

[0223] At step S163, study is performed using the pixel which constitutes the front are 
recording image memorized by the foreground are recording image storage section 44 in 
the study section 45 and a back are recording image, and the error image further 
memorized by the error image storage section 49 if needed, and, thereby, the prediction 
coefficient for predicting the pixel which constitutes the foreground of the 1st firame 
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thru/or each Nth frame is called for. 

[0224] Here, when study processing is first performed at step S163, since the error 
image is not memorized by the error image storage section 49, study is still performed in 
it, without using an error image (it being unable to use). 

[0225] The prediction coefficient obtained as a result of the study in step S163 is 
supplied to the prediction coefficient storage section 46 from the study section 45, and is 
memorized in the form to overwrite. If a prediction coefficient is memorized by the 
prediction coefficient storage section 46, it will set to step S164. The adaptation 
processing section 47 To the pixel which constitutes the front are recording image and 
back are recording image which were memorized by the foreground are recording image 
storage section 44, the prediction coefficient memorized by the prediction coefficient 
storage section 46, and a pan Adaptation processing which calculates the forecast of 
each pixel which constitutes the foreground of the 1st frame thru/or each Nth frame is 
performed by calculating a formula (8) and the linearity linear expression of (9) using 
the error image memorized by the error image storage section 49 if needed. 
[0226] Here, when adaptation processing is first performed at step S164, since the error 
image is not memorized by the error image storage section 49, adaptation processing is 
still performed in it, without using an error image (it being unable to use). 
[0227] The forecast of each pixel which constitutes the foreground of the 1st frame 
obtained as a result of the adaptation processing in step S164 thru/or each Nth frame is 
supphed to the error count section 48, in the error count section 48, in step S165, it is 
referring to the foreground of the 1st frame memorized by the foreground storage 
section 42 thru/or each Nth frame, and the prediction error of the forecast of each pixel 
which constitutes the foreground is searched for. 

[0228] And it progresses to step S166, and about each location of standard coordinates' 
existence within the limits, the absolute value of the prediction error in the location 
extracts the pixel which is max from the pixel which constitutes the foreground of the 
1st frame thru/or each Nth frame, and supplies the error count section 48 to the error 
image storage section 49 by using the image which becomes by the prediction error of 
the extracted pixel as an error image. In the error image storage section 49, the error 
image from the error count section 48 is memorized. 

[0229] Then, it progresses to step S167, total of the absolute value of the pixel value 
(prediction error) of each pixel which constitutes the error image memorized by the 
error image storage section 49 in the error judging section 50 is called for, and it is 
judged whether the total is below a predetermined threshold (following). 
[0230] In step S167, when it judges that total of the absolute value of the pixel value 
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(prediction error) of each pixel which constitutes an error image is not below a 
predetermined threshold (i.e., when the precision of the forecast of the foreground 
obtained from a prediction coefficient, and a front are recording image and a back are 
recording image (the need is accepted and it is an error image further) is not high), the 
same processing is repeated by step S163 return and the following. 
[0231] In addition, at next steps S163 or S164, since the error image is memorized by 
the error image storage section 49 in this case, as shown in drawing 29 , an error image 
besides a front are recording image and a back are recording image is also used, a 
prediction tap is constituted, and study processing or adaptation processing is 
performed, respectively. 

[0232] A prediction coefficient when it judges that total of the absolute value of the pixel 
value (prediction error) of each pixel which constitutes an error image is below a 
predetermined threshold in step S167 on the other hand, A front are recording image 
and a back are recording image (further) When the precision of the forecast of the 
foreground obtained from an error image if needed is high, it progresses to step S168. 
The error judging section 50 MUX51 is controlled, and the front are recording image 
memorized by the foreground are recording image storage section 44 and a back are 
recording image, the prediction coefficient memorized by the prediction coefficient 
storage section 46, and when the error image is memorized by the error image storage 
section 49, the error image is made to read and multiplex further. And MUX51 outputs 
the multiplexing data obtained as a result of the multiplexing to a multiplexer 17 
( drawing 2 ) as a coding result of the foreground of the 1st frame thru/or each Nth 
frame, and ends foreground coding processing. 

[0233] In addition, it may be made to make the new error image obtained whenever 
processing of step S166 is performed memorize in the form which overwrites the already 
memorized error image, and it leaves the already memorized error image to the error 
image storage section 49 as it is, and you may make it make it memorize it. 
[0234] When making a new error image memorize in the form which overwrites the 
error image already memorized by the error image storage section 49, although a 
certain amount of limitation is always in improvement in the precision of the forecast of 
a foreground since the number of error images is one even if study of a prediction 
coefficient is performed repeatedly, they can lessen the amount of data of multiplexing 
data. 

[0235] Although the amount of data of multiplexing data increases somewhat on the 
other hand since an error image becomes two or more sheets in leaving the error image 
already memorized by the error image storage section 49 in a new error image as it is 
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and making it memorize Since the error image of two or more sheets is used for a front 
are recording image and a back are recording image, and a list, a prediction tap is 
constituted and study processing or adaptation processing is performed, respectively as 
shown in drawing 30 , the precision of the forecast of a foreground can be raised more. 
In addition, drawing 30 shows the case where there are two error images. 
[0236] Moreover, although it asked for the prediction coefficient which calculates a 
forecast, also uses an error image when the prediction error is large, performs study 
processing again, and makes a prediction error small by performing adaptation 
processing using the prediction coefficient for which it asked by study processing in the 
above-mentioned case Irrespective of the size of a prediction error, it is also possible to 
make it output the prediction coefficient obtained as a result of study processing of the 
beginning by the study section 45 as a coding result of a foreground with the front are 
recording image and back are recording image which were memorized by the foreground 
are recording image storage section 44. In this case, the foreground coding section 16 is 
that the adaptation processing section 47, the error count section 48, the error image 
storage section 49, and the error judging section 50 can be constituted without 
preparing. 

[0237] Furthermore, in an above-mentioned case, a formula (9) defines the function g (F, 
B, E, n) which calculates the forecast of a foreground, and although it asked for the 
prediction coefficient of one set used common to this calculating the forecast of the 
foreground of the 1st frame thru/or all the Nth frame, in addition to this, a prediction 
coefficient can also be asked [ every frame and ] for every multiple frame. 
[0238] Namely, what is necessary is to define Function g (F, B, E, n), as shown in an 
equation (10), to build a normal equation for every frame, and just to make it ask for a 
prediction coefficient by solution Lycium chinense, in asking for a prediction coefficient 
for every frame. 
[02391 

g(F, B, E, n) =wFlnxfl+wF2nxf2-i- ... + wBlnxbl+wB2nxb2+ ... 
+wElnxel+wE2nxe2+ (10) 

Here, in a formula (lO), the prediction coefficient used for wFln, wF2n, wBln, 
wB2n, wEln, wE2n, and ... calculating the forecast of the foreground of the n-th 
frame is expressed. 

[0240] Moreover, although the forecast of a foreground was calculated by linearity 
primary prediction, the forecast of a foreground can also be made to ask more than by 
the secondary high order prediction type in addition to this here. 
[0241] Furthermore, although it was made to learn the prediction coefficient for 
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calculating the forecast of the foreground of the 1st frame thru/or each Nth frame here 
using the front are recording image and back are recording image which consisted of 
foregrounds of the 1st frame thru/or each Nth frame, it is also possible for it to be made 
to perform study of a prediction coefficient using images other than a front are 
recording image and a back are recording image. That is, study of a prediction 
coefficient can be made to be carried out by operating the pixel value of the image which 
becomes with the noise so that the prediction error of the forecast of a foreground may 
be made small using the image of one or more sheets which becomes with a noise. 
[0242] Next, drawing 31 shows the example of a configuration of the decoder 2 of 
drawing 1 . 

[0243] The coded data transmitted through a transmission medium 3 ( drawing 1 ) or 
the coded data reproduced from the record medium 4 ( drawing 1 ) is supplied to 
DMUX(demultiplexer) 81, and DMUX81 divides into a front are recording image, a back 
are recording image, a prediction coefficient, a whole background, and a camera motion 
vector the coded data supplied there. In addition, when an error image is contained in 
coded data, DMUX81 also separates the error image from coded data. 
[0244] Further, a front are recording image and a back are recording image, and when 
an error image is contained in coded data, the error image is supphed to the image 
storage section 86 from DMUX81. Moreover, a prediction coefficient, a whole 
background, or a camera motion vector is supplied to the prediction coefficient storage 
section 82, the background memory 87, and the camera motion vector storage section 88 
from DMUX81, respectively. 

[0245] The prediction coefficient storage section 82 memorizes the prediction coefficient 
from DMUX81. The adaptation processing section 83 calculates the forecast of the 
foreground of the 1st frame thru/or each Nth frame by the front are recording image 
memorized by the prediction coefficient memorized by the prediction coefficient storage 
section 82 and the list at the image storage section 86 and the back are recording image, 
and performing the same adaptation processing as a case [ in / using an error image / 
corresponding to the need further / the adaptation processing section 47 of drawing 24 ]. 
[0246] The foreground storage section 84 memorizes the forecast of the foreground of 
the 1st frame thru/or each Nth frame called for by the adaptation processing section 83 
as a decode result of the foreground of the 1st frame thru/or each Nth frame. 
[0247] The synthetic section 85 is based on the camera motion vector vn of the n-th 
frame memorized by the camera motion vector storage section 88 from the whole 
background memorized by the background memory 87. The background of the n-th 
frame is started (extracting), and by compounding the decode result of the background 
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of the n-th frame, and the foreground of the n-th frame memorized by the foreground 
storage section 84, the image of the n-th frame is decoded and it outputs. 
[0248] The image storage section 86 memorizes an error image in the front are 
recording image suppHed from DMUX81 and a back are recording image, and a Ust. The 
background memory 87 memorizes the whole background suppHed from DMUX81. The 
camera motion vector storage section 88 memorizes the camera motion vector of the 1st 
frame supphed from DMUX81 thru/or each Nth frame. 

[0249] Next, with reference to the flow chart of drawing 32 , the decode processing 
which decodes the image of the 1st frame thru/or the Nth frame as a series of images 
performed in the decoder 2 of drawing 31 is explained. 

[0250] First, in step S171, DMUX81 divides into a front are recording image, a back are 
recording image, a required error image, a prediction coefficient, a whole background, 
and a camera motion vector the coded data supplied there. A front are recording image, 
a back are recording image, and a required error image are supplied to the image 
storage section 86, and are memorized. Moreover, a prediction coefficient, a whole 
background, or a camera motion vector is supplied to the prediction coefficient storage 
section 82, the background memory 87, and the camera motion vector storage section 88, 
respectively, and is memorized. 

[025l] Then, it progresses to step S172, and using the prediction coefficient memorized 
by the prediction coefficient storage section 82, the front are recording image memorized 
by the list at the image storage section 86, a back are recording image, and a required 
error image, the adaptation processing section 83 is performing the same adaptation 
processing as the case in the adaptation processing section 47 of drawing 24 , and 
calculates the forecast of the foreground of the 1st frame thru/or each Nth frame. This 
forecast is supplied to the foreground storage section 84, and is memorized as a decode 
result of the foreground of the 1st frame thru/or each Nth frame. 
[0252] And it progresses to step S173, and in the synthetic section 85, based on the 
camera motion vector vn of the n-th frame memorized by the camera motion vector 
storage section 88, the background of the n-th frame is started from the whole 
background memorized by the background memory 87, and the decode result of the 
background of the n-th frame and the foreground of the n-th frame memorized by the 
foreground storage section 84 is compounded. In the synthetic section 85, the above 
processing is performed about the 1st frame thrvi/or all the Nth frame, and ends decode 
processing. 

[0253] Next, hardware can also perform a series of processings mentioned above, and 
software can also perform. When software performs a series of processings, the program 
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which constitutes the software is installed in the computer built into the encoder 1 and 
decoder 2 as hardware of dedication, or the general-purpose computer which performs 
various kinds of processings by instalHng various kinds of programs. 
[0254] Then, the medium used in order to install in a computer the program which 
performs a series of processings mentioned above and to make it into the condition 
which can be performed by computer with reference to drawing 33 is explained. 
[0255] As shown in drawing 33 (A), a user can be provided with a program in the 
condition of having installed in the hard disk 102 and semiconductor memory 103 as a 
record medium which are built in the computer 101 beforehand. 

[0256] Or as shown in drawing 33 (B), a program can be stored in record media, such as 
a floppy disk 111, CD-ROM (Compact Disc Read Only Memory)ll2, the MO (Magneto 
optical) disk 113, DVD (Digital Versatile Disc)ll4, a magnetic disk 115, and 
semiconductor memory 116, temporarily or permanently, and can be offered as a 
software package again. 

[0257] Fvirthermore, it transmits to a computer 101 on radio, or a program is 
transmitted to a computer 123 with a cable through the networks 131, such as LAN 
(Local Area Network) and the Internet, and can be made to store in the hard disk 102 to 
build in in a computer 101 through the satellite 122 for digital satellite broadcasting 
services from the download site 121, as shown in drawing 33 (C). 

[0258] The medium in this description means the concept of the wide sense containing 
all these media. 

[0259] Moreover, it is not necessary to necessarily process the step which describes the 
program offered by the medium to time series in accordance with the sequence indicated 
as a flow chart, and it is a juxtaposition thing also including the processing (for example, 
parallel processing or processing by the object) performed according to an individual in 
this description. 

[0260] Next, drawing 34 shows the example of a configuration of the computer 101 of 

drawing 33 . 

[0261] The computer 101 contains CPU (Central Processing Unit)l42, as shown in 
drawing 34 . The input/output interface 145 is connected to CPU142 through the bus 
141, and CPU142 will perform the program stored in ROM (Read Only Memory) 143 
corresponding to the semiconductor memory 103 of drawing 33 (A) according to it, if a 
command is inputted when the input section 147 which consists of a keyboard, a mouse, 
etc. is operated by the user through an input/output interface 145. Or it is transmitted 
from the program and satellite 122 with which CPU142 is stored in the hard disk 102 
again, or a network 131, and the program which reading appearance was carried out 
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from the program which was received in the communications department 148 and 
installed on the hard disk 102 or the floppy disk 111 with which the drive 149 was 
equipped, CD-R0M112, MO disk 113, DVD114, or the magnetic disk 115, and was 
installed on the hard disk 102 is loaded to RAM (Random Access Memory) 144, and is 
performed. And CPU142 outputs the processing result to the display 146 which consists 
of LCD (Liquid CryStal Display) etc. through an input/output interface 145 if needed. 
[0262] 

[Effect of the Invention] According to the medium, the pixel which is in the same 
location spatially where alignment of the background of each of that screen is performed 
is detected by the image processing system of this invention and the image-processing 
approach, and the list from a series of images, and counting of the frequency of the pixel 
value of the pixel which is in the same location spatially is carried out to them. And 
based on the background pixel the rate of frequency to the number of the pixel which is 
in the same location spatially was remembered to be by background-image storage 
means to memorize a background image, as a pixel value of the background pixel from 
which the pixel value beyond a predetermined value constitutes a background image, 
and the pixel value was remembered to be, the background image memorized by the 
background-image storage means is extended. Therefore, it becomes possible from a 
series of images to extract a background with a sufficient precision. 



DESCRIPTION OF DRAWINGS 



[Brief Description of the Drawings] 

[Drawing 1] It is drawing showing the example of a configuration of the gestalt of 1 
operation of the picture transmission equipment which applied this invention. 
[Drawing 2] It is the block diagram showing the example of a configuration of the 
encoder 1 of drawing 1 , 

[Drawing 3] It is drawing showing a series of images memorized by the are recording 
section 1 1 of drawing 2 . 

[Drawing 4] It is drawing for explaining processing of the camera motion detecting 
element 1 2 of drawing 2 . 

[Drawing 5] It is the block diagram showing the example of a configuration of the 
camera motion detecting element 12 of drawing 2 . 

[Drawing 6] It is drawing for explaining how asking for the camera motion vector by the 
camera motion detecting element 1 2 of drawing 5 . 
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[Drawing 7] It is a flow chart for explaining processing (camera motion detection 
processing) of the camera motion detecting element 1 2 of drawing 5 . 
[Drawing 8] It is a flow chart for explaining the 1st example of the detail of the 
processing (center-of-gravity calculation processing of an attention frame) in step S3 
of drawing 7 . 

[Drawing 9] It is drawing showing a level table. 

[Drawing 10] It is a flow chart for explaining the detail of the processing 
(center-of-gravity calculation processing of the motion detection range) in step S4 of 
drawing 7 . 

[Drawing 1 1] It is a flow chart for explaining the 2nd example of the detail of the 

processing in step S3 of drawing 7 . 

[Drawing 12] It is drawing showing a class table. 

[Drawing 13] It is a flow chart for explaining the 3rd example of the detail of the 

processing in step S3 of drawing 7 . 

[Drawing 14] It is drawing for explaining an edge pixel. 

[Drawing 15] It is a flow chart for explaining the detail of processing (motion-less 
judgment processing) of the motion-less judgment section 25 of drawing 5 . 
[Drawing 16] It is the block diagram showing the example of a configuration of the 
background extract section 14 of drawing 2 . 

[Drawing 17] It is drawing for explaining processing of the drawing 16's existence 
range detecting element 31. 

[Drawing 18] It is drawing for explaining processing of the read-out section 32 of 

drawing 1 6 . 

[Drawing 19] It is drawing showing a frequency table. 

[Drawing 20] It is a flow chart for explaining processing (background extract 
processing) of the background extract section 14 of drawing 1 6 . 
[Drawing 21] It is a flow chart for explaining the detail of processing (background pixel 
extract processing) of step SI 24 of drawing 20 . 

[Drawing 22] It is a flow chart for explaining the detail of processing (background 
escape processing) of step SI 25 of drawing 20 . 

[Drawing 23] It is drawing for explaining background escape processing of drawing 22 . 
[Drawing 24] It is the block diagram showing the example of a configuration of the 
foreground coding section 1 6 of drawing 2 . 

[Drawing 25] It is drawing for explaining processing of the foreground are recording 
image configuration section 43 of drawing 24 . 

[Drawing 26] It is the block diagram showing the example of a configuration of the 
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study section 45 of drawing 24 . 

[Drawing 27] It is the block diagram showing the example of a configuration of the 
adaptation processing section 47 of drawing 24 . 

[Drawing 28] It is a flow chart for explaining the detail of processing (foreground 
coding processing) of the foreground coding section 16 of drawing 24 . 
[Drawing 29] A prediction tap is drawing showing signs that it is constituted using an 
error image. 

[Drawing 30] A prediction tap is drawing showing signs that it is constituted using the 
error image of two sheets. 

[Drawing 31] It is the block diagram showing the example of a configuration of the 
decoder 2 of drawing 1 . 

[Drawing 32] It is a flow chart for explaining processing (decode processing) of the 
decoder 2 of drawing 31 . 

[Drawing 33] It is drawing for explaining the medium which applied this invention. 
[Drawing 34] It is the block diagram showing the example of a configuration of the 
computer 101 of drawing 33 . 
[Description of Notations] 

1 Encoder 2 Decoder, 3 Transmission medium 4 A record medium, 1 1 are-recording 
section 1 2 A camera motion detecting element, 1 3 The camera motion vector storage 
section and 14 background extract section 15 Background memory, 16 Foreground 
coding section 17 MUX, 21 Center-of-gravity calculation section 22 A vector 
detecting element, 23 A write-in control section, 24 An are recording image memory, 
25 Motion-less judgment section 31 An existence range detecting element, 32 
Read-out section 33 Frequency count area 34 The frequency table storage section 
and 35 The write-in section 36 Background flag memory 41 The foreground extract 
section, 42 foreground storage section 43 The foreground are recording image 
configuration section, 44 foreground are recording image storage section, 45 The 
study section, 46 Prediction coefficient storage section 47 The adaptation processing 
section, 48 Error count section 49 The error image storage section, 50 Error judging 
section 51 MUX, 61 prediction tap configuration section 62 The normal equation 
configuration section, 63 The prediction coefficient calculation section, 71 The 
prediction tap configuration section, 72 Prediction operation part 81 DMUX, 82 
Prediction coefficient storage section 83 The adaptation processing section, 84 
Foreground storage section 85 The synthetic section, 86 image storage section 87 
Background memory 88 The camera motion vector storage section, 101 Computer 
102 Hard disk 103 Semiconductor memory, 1 1 1 Floppy disk 112 CD-ROM, 1 13 An MO 
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disk, 114 DVD, 115 Magnetic disk 116 Semiconductor memory, 121 download site 122 
satellites, 131 Network 141 buses 142 CPU 143 ROM 144 RAM 145 Input/output 
interface 146 Display 147 Input section 148 Communications department 149 Drive 
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